Data balancing method and device, equipment and storage medium

Abstract

The invention relates to a data balancing method and device, equipment and a storage medium, and the method comprises the steps: determining a transfer node meeting a high-load operation condition according to a performance index and a first fragment number corresponding to each data node; in the fragments included in the transfer node, whether a first fragment set meeting the high read-write operation condition exists or not is detected, and each fragment in the first fragment set corresponds to a first preset label; when the first fragment set meeting the high read-write operation condition exists, according to a first preset label, determining a target node meeting a low load operation condition and meeting the low read-write operation condition under the first preset label; and according to the first fragment set and the target node, generating a data transfer task to transfer the fragments in the first fragment set to the target node. According to the method, the fragments with large read / write traffic can be prevented from being accumulated at a certain data node, so that the influence on the throughput and availability of the whole cluster is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data balancing method, device, equipment and storage medium. Background Art

[0002] A distributed database is a database cluster that distributes the data and management functions of a database management cluster across multiple physical nodes. When storing a data table, the cluster first splits the data table into multiple shards, and then distributes these shards to each data node in the cluster.

[0003] In practice, since the shards are unevenly distributed in the cluster, the cluster also needs to use a data balancing algorithm to evenly distribute the shards among the data nodes. The data balancing algorithm determines the data node with the most shards among these data nodes and transfers the shards in this data node to other data nodes.

[0004] However, the above data balancing algorithm is designed under the premise that the read / write traffic carried by each shard is roughly equal. However, this assumption is often not true in actual situations. Once shards with large read / write traffic accumulate on a certain data node, the processing capacity of a single data node is limited, which affects the throughput and availability of the entire cluster. Summary of the invention

[0005] The present application provides a data balancing method, apparatus, device and storage medium, which can prevent shards with large read / write traffic from piling up on a certain data node, thereby avoiding impact on the throughput and availability of the entire cluster.

[0006] In a first aspect, the present application provides a data balancing method, the method comprising:

[0007] Obtain the performance indicators corresponding to each data node in the cluster and the number of first shards of the shards included in each data node;

[0008] Determine a transfer node that meets the high-load operation condition according to the performance index corresponding to each data node and the number of first shards;

[0009] Among the shards included in the transfer node, detecting whether there is a first shard set that meets the high read / write operation condition, each shard in the first shard set corresponding to a first preset label;

[0010] When there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label;

[0011] A data transfer task is generated according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

[0012] Optionally, determining the transfer node that meets the high-load operation condition according to the performance indicator corresponding to each data node and the number of first shards includes:

[0013] For each data node, obtain a first parameter value corresponding to the data node, and determine a first value corresponding to the data node according to the first number of shards corresponding to the data node and the first parameter value, where the first parameter value indicates a capability of the data node to store data;

[0014] Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value;

[0015] Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node;

[0016] When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

[0017] Optionally, determining the first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value includes:

[0018] For each data node, a performance indicator corresponding to the data node is multiplied by an indicator weight to obtain a second value corresponding to the performance indicator, and a sum of the second value and the first value is determined as a first load value of the data node.

[0019] Optionally, detecting whether there is a first set of shards meeting the high read / write operation condition among the shards included in the transfer node includes:

[0020] Obtain a second load value corresponding to each second shard set on the transfer node, wherein the read and write heat labels corresponding to each shard in the second shard set are the same;

[0021] In the second shard set, detecting whether there is a third shard set corresponding to a first preset tag, where the first preset tag is a tag indicating a high read / write frequency;

[0022] When there is a third shard set corresponding to the first preset label, the second average load value corresponding to the first preset label is determined according to the first preset label and the correspondence between the read-write heat label and the average load value; when the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read-write operating conditions.

[0023] Optionally, the method further includes:

[0024] For each data node, obtain the read-write heat label of each shard on the data node, divide the shards corresponding to the same read-write heat label into the same set to obtain at least one fourth shard set, obtain the number of second shards corresponding to each fourth shard set, determine the third load value corresponding to each fourth shard set based on the number of second shards corresponding to each fourth shard set and the first parameter value of the data node, and establish a correspondence between the data node, the shard set and the load value based on the data node, the fourth shard set on the data node and the third load value corresponding to each fourth shard set.

[0025] Optionally, the method further includes:

[0026] For each read / write heat label, determine the total number of shards corresponding to the read / write heat label in the cluster, and obtain the total number corresponding to the read / write heat label;

[0027] Determine a third average load value corresponding to each read / write heat label according to the total number corresponding to each read / write heat label and a second parameter corresponding to the cluster, wherein the second parameter indicates a data storage capability of the cluster;

[0028] According to each read / write heat label and the corresponding third average load value, a corresponding relationship between the read / write heat label and the average load value is established.

[0029] Optionally, the method further includes:

[0030] Determine the target business to which the slice to be marked belongs;

[0031] According to the target hotspot detection rule corresponding to the target business, detect whether the shard to be marked is a hotspot shard;

[0032] When the shard to be marked is a hotspot shard, obtaining a second preset tag corresponding to the target service, and setting the second preset tag as a read-write heat tag of the shard to be marked;

[0033] When the shard to be marked is a non-hotspot shard, a preset third preset tag is obtained, and the third preset tag is set as the read-write heat tag of the shard to be marked.

[0034] In a second aspect, the present application provides a data balancing device, the device comprising:

[0035] An acquisition unit, used to acquire a performance indicator corresponding to each data node in the cluster and the number of first shards of shards included in each data node;

[0036] A first determining unit, configured to determine a transfer node that meets a high-load operation condition according to a performance indicator corresponding to each data node and the number of first shards;

[0037] A detection unit, configured to detect whether there is a first set of shards that meet the high read / write operation condition in the shards included in the transfer node, each shard in the first set of shards corresponding to a first preset label;

[0038] A second determining unit is configured to determine, when there is a first shard set that meets the high read / write operation condition, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label according to the first preset label;

[0039] A generating unit is used to generate a data transfer task according to the first shard set and the target node, so as to transfer the shards in the first shard set to the target node.

[0040] Optionally, the first determining unit is configured to:

[0041] For each data node, obtain a first parameter value corresponding to the data node, and determine a first value corresponding to the data node according to the first number of shards corresponding to the data node and the first parameter value, where the first parameter value indicates a capability of the data node to store data;

[0042] Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value;

[0043] Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node;

[0044] When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

[0045] Optionally, the first determining unit is configured to:

[0046] For each data node, a performance indicator corresponding to the data node is multiplied by an indicator weight to obtain a second value corresponding to the performance indicator, and a sum of the second value and the first value is determined as a first load value of the data node.

[0047] Optionally, the detection unit is used to:

[0048] Obtain a second load value corresponding to each second shard set on the transfer node, wherein the read and write heat labels corresponding to each shard in the second shard set are the same;

[0049] In the second shard set, detecting whether there is a third shard set corresponding to a first preset tag, where the first preset tag is a tag indicating a high read / write frequency;

[0050] When there is a third shard set corresponding to the first preset label, the second average load value corresponding to the first preset label is determined according to the first preset label and the correspondence between the read-write heat label and the average load value; when the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read-write operating conditions.

[0051] Optionally, the device further includes a first establishing unit, wherein the first establishing unit is configured to:

[0052] For each data node, obtain the read-write heat label of each shard on the data node, divide the shards corresponding to the same read-write heat label into the same set to obtain at least one fourth shard set, obtain the number of second shards corresponding to each fourth shard set, determine the third load value corresponding to each fourth shard set based on the number of second shards corresponding to each fourth shard set and the first parameter value of the data node, and establish a correspondence between the data node, the shard set and the load value based on the data node, the fourth shard set on the data node and the third load value corresponding to each fourth shard set.

[0053] Optionally, the device further includes a second establishing unit, wherein the second establishing unit is configured to:

[0054] For each read / write heat label, determine the total number of shards corresponding to the read / write heat label in the cluster, and obtain the total number corresponding to the read / write heat label;

[0055] Determine a third average load value corresponding to each read / write heat label according to the total number corresponding to each read / write heat label and a second parameter corresponding to the cluster, wherein the second parameter indicates a data storage capability of the cluster;

[0056] According to each read / write heat label and the corresponding third average load value, a corresponding relationship between the read / write heat label and the average load value is established.

[0057] Optionally, the device further comprises a marking unit, wherein the marking unit is used to:

[0058] Determine the target business to which the slice to be marked belongs;

[0059] According to the target hotspot detection rule corresponding to the target business, detect whether the shard to be marked is a hotspot shard;

[0060] When the shard to be marked is a hotspot shard, obtaining a second preset tag corresponding to the target service, and setting the second preset tag as a read-write heat tag of the shard to be marked;

[0061] When the shard to be marked is a non-hotspot shard, a preset third preset tag is obtained, and the third preset tag is set as the read-write heat tag of the shard to be marked.

[0062] In a third aspect, the present application provides a data balancing device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0063] Get the performance indicators corresponding to each data node in the cluster and the number of first shards of the included shards;

[0064] Determine a transfer node that meets the high-load operation condition according to the performance index corresponding to each data node and the number of first shards;

[0065] Among the shards included in the transfer node, detecting whether there is a first shard set that meets the high read / write operation condition, each shard in the first shard set corresponding to a first preset label;

[0066] When there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label;

[0067] A data transfer task is generated according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data balancing method when executed by a processor.

[0069] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: In the embodiment of the present application, the performance index corresponding to each data node in the cluster and the first number of shards included in each data node are obtained; according to the performance index corresponding to each data node and the first number of shards, a transfer node that meets the high-load operation condition is determined; in the shards included in the transfer node, it is detected whether there is a first shard set that meets the high read / write operation condition, and each shard in the first shard set corresponds to a first preset label; when there is a first shard set that meets the high read / write operation condition, according to the first preset label, a target node that meets the low-load operation condition and meets the low read / write operation condition under the first preset label is determined; according to the first shard set and the target node, a data transfer task is generated to transfer the shards in the first shard set to the target node. It can be seen that the present application can transfer the transfer shards in the transfer shard set that meets the high read / write operation condition to other nodes, avoiding the accumulation of shards with large read / write traffic at a certain node, thereby avoiding the impact on the throughput and availability of the entire cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0073] Figure 1 A flowchart of a data balancing method provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of a flow chart of a transfer node determination method provided in an embodiment of the present application;

[0075] Figure 3 A flow chart of a load value calculation method provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of a method for determining a read-write thermal tag provided in an embodiment of the present application;

[0077] Figure 5 A flowchart of a method for determining a corresponding relationship provided in an embodiment of the present application;

[0078] Figure 6 A flowchart of a method for determining a first shard set provided in an embodiment of the present application;

[0079] Figure 7 A flowchart of another data balancing method provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of a flow chart of a data equalization device provided in an embodiment of the present application;

[0081] Fig. 9 A schematic diagram of a data equalization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0083] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0084] A distributed database is a database cluster that distributes the data and management functions of a database management cluster on multiple physical nodes. When the cluster stores a data table, it first divides the data table into multiple shards, and distributes these shards to each data node in the cluster. In practice, since the shards are unevenly distributed in the cluster, the cluster also needs to use a data balancing algorithm to make the shards evenly distributed among these data nodes. The data balancing algorithm determines the data node with the most shards among these data nodes, and transfers the shards in the data node to other data nodes. However, the above data balancing algorithm is designed under the premise that the read / write traffic carried by each shard is roughly equal. However, this assumption is often not true in actual situations. Once shards with large read / write traffic accumulate on a certain data node, the processing capacity of a single data node is limited, which affects the throughput and availability of the entire cluster.

[0085] Based on the above, the embodiment of the present application provides a data balancing method, which can prevent the accumulation of shards with large read / write traffic on a certain data node, thereby avoiding the impact on the throughput and availability of the entire cluster. Figure 1 As shown, the specific steps include:

[0086] Step 101, obtaining the performance indicator corresponding to each data node in the cluster and the number of first shards of the shards included in each data node.

[0087] The cluster includes management nodes and data nodes. The management node is a node that manages the data node, and the data node is a node that stores data and improves data reading and writing services. Each data node is set with multiple shards. These shards may correspond to the same business or different businesses. The data corresponding to the same business belongs to the same logical table. The logical table is a table structure with a certain logical relationship. For example, data node 1 includes shards corresponding to business 1 and shards corresponding to business 2, and data node 2 includes shards corresponding to business 1. Performance indicators are used to measure specific parameters of data nodes in terms of performance, such as CPU utilization, memory utilization, etc.

[0088] In this step, when the management node needs to load balance the cluster, the management node can send a data request to each data node in the cluster so that each data node returns the required data to the control node. After the data node receives the data request, the data node sends the data required by the management node to the management node. Or each data node in the cluster periodically sends the data it needs to the management node. For example, when the management node needs to load balance the cluster, the management node can send a data request to each data node in the cluster. After that, each data node sends its own performance indicators and the first number of shards to the management node. Or, each data node in the cluster periodically sends its own performance indicators and the first number of shards to the management node.

[0089] It should be noted that the management node will receive data sent by multiple data nodes. In order to distinguish the data sent by each data node, the data node identifier can also be used to uniquely identify the data sent by the data node, that is, a corresponding relationship between the data node identifier corresponding to each data node and the data it sends is established to determine the data node to be operated based on the data node identifier.

[0090] Step 102: Determine a transfer node that meets the high-load operation condition according to the performance indicator corresponding to each data node and the number of first shards.

[0091] In this step, when the number of shards included in a data node is larger and the performance of the data node is worse, the load of the data node is larger, and when the number of shards included in a data node is smaller and the performance of the data node is better, the load of the data node is smaller. Therefore, for each data node, the management node needs to determine the first load value corresponding to the data node based on the performance index corresponding to the data node and the first number of shards. When the first load value of a data node is greater than the first average load value, it is determined that the data node meets the high-load operation condition and the data node is determined as a transfer node.

[0092] Step 103: Detect whether there is a first set of shards that meets the high read / write operation condition among the shards included in the transfer node.

[0093] Each shard in the first shard set corresponds to a first preset tag, and the first preset tag is a tag indicating a high read and write frequency.

[0094] In this step, each shard in the transfer node corresponds to a read / write heat label, so the shards in the transfer node can be divided according to the read / write heat label corresponding to each shard to obtain at least one second shard set. In the at least one second shard set, it is detected whether there is a first shard set corresponding to the first preset label to determine whether there is a first shard set that meets the high read / write operation condition.

[0095] Step 104 , when there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label.

[0096] In this step, when there is a first shard set that meets the high read / write operating condition, a target node that meets the low load operating condition and meets the low read / write operating condition under the first preset label can be determined using the first preset label.

[0097] Further, the step of determining the target node is: the first load value and the first average load value of each data node can be obtained, and among all data nodes, the data node corresponding to the first load value less than the first average load value is determined as a reference node of low-load operation condition. Then, the reference nodes are sorted according to the first load value corresponding to each reference node, and then the following operations are performed on the reference nodes in order from small to large: according to the first preset label, in the shards in the reference node, the shard set corresponding to the first preset label is determined, and according to the correspondence between the shard set and the preset shard set and the load value, the second load value corresponding to the shard set is determined, that is, the second load value corresponding to the reference node under the first preset label. Afterwards, according to the first preset label, the corresponding relationship between the read-write heat label and the average load value, the second average load value corresponding to the cluster under the first preset label is determined, and when the second load value is less than the second average load value, the reference node is determined as the target node. When the second load value is not less than the second average load value, the next reference node is detected.

[0098] It should be noted that in the scenario of shard splitting, the target node that meets the low-load operating conditions and the preset operating conditions under the read-write heat label corresponding to the shard can also be determined, and the shard obtained by the split can be transferred to the target node. When the read-write heat label is the first preset label, it is the target node that meets the low read-write operating conditions under the first preset label. When the read-write heat label is the third preset label, it is the target node that meets the low-load operating conditions under the third preset label. The third preset label is used to indicate a label with low read and write frequencies.

[0099] Step 104: Generate a data transfer task according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

[0100] In this step, according to a preset rule, a shard is selected from the first shard set, and a data transfer task is generated according to the shard and the target node to transfer the shard to the target node.

[0101] Furthermore, the preset rule may be to randomly select a shard, or to select the earliest or latest shard, or other shard selection rules, which are not limited here.

[0102] In an embodiment of the present application, the performance index corresponding to each data node in the cluster and the first number of shards included in each data node are obtained; based on the performance index corresponding to each data node and the first number of shards, a transfer node that meets the high-load operation condition is determined; among the shards included in the transfer node, it is detected whether there is a first shard set that meets the high read / write operation condition, and each shard in the first shard set corresponds to a first preset label; when there is a first shard set that meets the high read / write operation condition, according to the first preset label, a target node that meets the low-load operation condition and meets the low read / write operation condition under the first preset label is determined; based on the first shard set and the target node, a data transfer task is generated to transfer the shards in the first shard set to the target node. It can be seen that the present application can transfer the transfer shards in the transfer shard set that meets the high read / write operation condition to other nodes, so as to avoid the accumulation of shards with large read / write traffic at a certain node, thereby avoiding the impact on the throughput and availability of the entire cluster.

[0103] In the embodiment of the present application, the transfer node that meets the high-load operation condition can also be determined according to the performance index corresponding to each data node and the number of first shards. Therefore, the embodiment of the present application provides a transfer node determination method, such as Figure 2 As shown, the specific steps include:

[0104] Step 201: For each data node, obtain a first parameter value corresponding to the data node, and determine a first value corresponding to the data node according to the first shard quantity and the first parameter value corresponding to the data node.

[0105] Among them, the first parameter value is used to indicate the data node's ability to store data, which is related to the data node's ability to store data and performance indicators. The parameter value is a value set by the technician based on the data node's ability to store data.

[0106] In this step, the management node stores the correspondence between data nodes and parameter values. When the parameter value of a data node needs to be obtained, the first parameter value corresponding to the data node can be determined based on the data node and the pre-set correspondence. Then, the first shard number corresponding to the data node is divided by the first parameter value to obtain the first numerical value corresponding to the data node.

[0107] For example, the first value is the first number of shards / the first parameter value.

[0108] Step 202: Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value.

[0109] The first load value is used to indicate the load condition of the data node. When the load value of a data node is larger, the load of the data node is larger, and when the load value of the data node is smaller, the load of the data node is smaller.

[0110] In this step, the management node may determine the second value corresponding to each data node according to the performance indicator corresponding to each data node, and then add the first value and the second value of each data node to obtain the first load value of each data node.

[0111] Step 203: Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node.

[0112] The first average load value is used to indicate the load condition of the cluster. When the load value of a cluster is larger, the load of the cluster is larger, and when the load value of the cluster is smaller, the load of the cluster is smaller.

[0113] In this step, the management node can obtain the first load value corresponding to each data node and the total number of data nodes, then add these first load values ​​to obtain the sum of the first load values, divide the sum by the total number, and obtain the first average load value corresponding to the cluster.

[0114] Step 204: When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

[0115] In this step, when the first load value of a data node is greater than the first average load value, it means that the load of the data node is relatively large and the shards in the data node need to be transferred. Therefore, it is necessary to detect whether there are data nodes in the cluster whose corresponding first load values ​​are greater than the first average load value. When there are data nodes whose corresponding first load values ​​are greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation conditions.

[0116] In the embodiment of the present application, for each data node, the first load value corresponding to the data node can also be calculated according to the performance indicator corresponding to the data node and the first value. Therefore, the embodiment of the present application provides a load value calculation method, such as Figure 3 As shown, the specific steps include:

[0117] Step 301: multiply the performance indicator corresponding to the data node by the indicator weight to obtain a second value corresponding to the performance indicator.

[0118] Among them, the indicator weight is set by the technician according to the impact of the performance indicator on the load. When the impact of a certain performance indicator on the load is greater, the corresponding indicator weight is greater; when the impact of a certain performance indicator on the load is smaller, the corresponding indicator weight is smaller.

[0119] In this step, the performance index corresponding to the data node is obtained, and the performance index is multiplied by the index weight to obtain a second value corresponding to the performance index.

[0120] For example, the performance indicators include cpu usage cpu_metrics, memory usage mem_metrics, disk space usage disk_space_metrics, and QPS (Queries Per Second) qps_metrics, where the indicator weight corresponding to cpu usage is cpu_weight, the indicator weight corresponding to memory usage is mem_weight, the indicator weight corresponding to disk space usage is disk_space_weight, and the indicator weight corresponding to QPS is qps_weight. The second value corresponding to cpu usage is cpu_metrics×

[0121] cpu_weight, the second value corresponding to memory usage is mem_metrics×

[0122] mem_weight, the second value corresponding to the disk space usage is disk_space_metrics×disk_space_weight, and the second value corresponding to QPS is qps_metrics×qps_weight.

[0123] Step 302: determine the sum of the second value and the first value as the first load value of the data node.

[0124] For example, the first load value=the first number of shards / the first parameter value+cpu_metrics×cpu_weight+mem_metrics×mem_weight+disk_space_metrics×disk_space_weight+qps_metrics×qps_weight.

[0125] In the embodiment of the present application, the shards in the data node are divided according to the read-write heat label corresponding to each shard. Therefore, before dividing the shards in the data node, it is also necessary to determine the read-write heat label corresponding to each shard. Based on this, the embodiment of the present application provides a method for determining the read-write heat label. Figure 4 As shown, the specific steps include:

[0126] Step 401: determine the target service to which the slice to be marked belongs.

[0127] In this step, when some shards correspond to the same business, these shards belong to the same logical table, and different logical tables use different heat detection rules. Therefore, when it is necessary to determine the label of the shard to be marked, the target business to which the shard to be marked belongs is determined based on the shard to be marked and the correspondence between the shard and the business.

[0128] Step 402: According to the target hotspot detection rule corresponding to the target service, detect whether the slice to be marked is a hotspot slice.

[0129] Among them, hotspot shards are shards with high read and write frequencies.

[0130] In this step, according to the target business and the correspondence between the business and the hotspot detection rule, the target hotspot detection rule corresponding to the target business is determined, and the target hotspot detection rule is used to detect whether the shard to be marked is a hotspot shard.

[0131] When the logical table corresponding to the business is a logical table whose key starts with a date, the read and write traffic is generally concentrated in the shard corresponding to the most recent date. Therefore, it is necessary to detect whether the date of the shard to be marked is within the most recent date. If so, the shard to be marked is determined to be a hot shard. If not, the shard to be marked is not a hot shard. When the logical table corresponding to the business is a logical table whose key starts with user_id, the read and write traffic will be concentrated on the shard corresponding to the large customer. Therefore, obtain the user corresponding to the shard to be marked and detect whether the user is a preset user. If so, the shard to be marked is determined to be a hot shard. If not, the shard to be marked is not a hot shard.

[0132] Step 403: When the shard to be marked is a hotspot shard, obtain a second preset tag corresponding to the target service, and set the second preset tag as the read / write heat tag of the shard to be marked.

[0133] The second preset tag is a tag corresponding to the target service and used to indicate a high read / write frequency. Different services correspond to different preset tags, so that different services can be distinguished based on different preset tags.

[0134] In this step, when the shard to be marked is a hotspot shard, it means that the shard to be marked is a shard with high read and write frequency. In order to achieve uniform sharding of the shards corresponding to each business in the cluster, the second preset label corresponding to the target business is determined according to the target business and the correspondence between the business and the label, and the second preset label is set as the read and write heat label of the shard to be marked.

[0135] It should be noted that, in one embodiment, it is not necessary to distinguish services based on tags. That is, when the shard to be marked is a hot shard, a preset fourth preset tag is obtained, and the fourth preset tag is set as the read-write heat tag of the shard to be marked. The fourth preset tag is used to indicate a tag with high read-write frequency.

[0136] Step 404: when the shard to be marked is a non-hotspot shard, obtain a preset third preset tag, and set the third preset tag as the read / write heat tag of the shard to be marked.

[0137] The third preset tag is used to indicate a tag with a low reading and writing frequency.

[0138] In this step, when the shard to be marked is a non-hotspot shard, it means that the shard to be marked is a shard with low read / write frequency. Since the shard with low read / write frequency is hardly read / written, the shard only needs to be stored, and there is no need to further distinguish the services of these shards. Therefore, the preset third preset label is obtained, and the third preset label is set as the read / write heat label of the shard to be marked.

[0139] In the embodiment of the present application, before detecting the first shard set, it is also necessary to create a corresponding relationship between the read / write heat label and the average load value, and then based on the corresponding relationship, detect whether the first shard set exists. Therefore, the embodiment of the present application provides a corresponding relationship determination method, such as Figure 5 As shown, the specific steps include:

[0140] Step 501: For each read / write heat tag, determine the total number of shards corresponding to the read / write heat tag in the cluster to obtain the total number corresponding to the read / write heat tag.

[0141] In this step, for a certain read-write heat label, in the cluster, determine the number of shards corresponding to the read-write heat label in each data node, add up these shard numbers, and get the total number of shards corresponding to the read-write heat label, and determine the total number as the total number corresponding to the read-write heat label. Based on the above method, the total number corresponding to all read-write heat label shards is obtained.

[0142] Step 502: Determine a third average load value corresponding to each read / write heat tag according to the total number corresponding to each read / write heat tag and the second parameter corresponding to the cluster.

[0143] The second parameter indicates the data storage capacity of the cluster, and the second parameter value may be the sum of the first parameter values ​​corresponding to all data nodes in the cluster. The third average load value is used to indicate the load of the shards corresponding to the same read / write heat label in the cluster.

[0144] In this step, for each read / write heat tag, the total number corresponding to the read / write heat tag is divided by the second parameter corresponding to the cluster to obtain a third average load value corresponding to the read / write heat tag.

[0145] Step 503: Establish a correspondence between the read / write heat label and the average load value according to each read / write heat label and the corresponding third average load value.

[0146] In this step, a correspondence between the read / write heat tag and the average load value is established according to each read / write heat tag and the third average load value corresponding to each read / write heat tag, and the correspondence is stored.

[0147] It should be noted that the present application can also determine the load value of each data node under a certain read-write heat label, and calculate the average of these load values ​​to obtain the third average load value corresponding to the read-write heat label. Based on the above method, the third average load value corresponding to each read-write heat label is obtained. Afterwards, according to each read-write heat label and the third average load value corresponding to each read-write heat label, a corresponding relationship between the read-write heat label and the average load value is established.

[0148] In the embodiment of the present application, at least one second shard set can be obtained for the shards included in the transfer node according to the read / write heat label corresponding to each shard, and then in the second shard set, it is detected whether there is a first shard set that meets the high read / write operation condition. Therefore, the embodiment of the present application provides a method for determining the first shard set, such as Figure 6 As shown, the specific steps include:

[0149] Step 601, obtaining a second load value corresponding to each second shard set on the transfer node.

[0150] Among them, the read and write heat labels corresponding to each shard in the second shard set are the same.

[0151] In this step, the shard set on the transfer node is taken as the second shard set, and the second load value corresponding to each second shard set on the transfer node is determined in the pre-stored correspondence between the data node, shard set and load value according to the transfer node and each second shard set.

[0152] Before executing this step, for each data node, obtain the read-write heat label of each shard on the data node, divide the shards corresponding to the same read-write heat label into the same set, obtain at least one fourth shard set, obtain the number of second shards corresponding to each fourth shard set, divide the number of second shards corresponding to each fourth shard set by the first parameter value of the data node, obtain the third load value corresponding to each fourth shard set, and establish a correspondence between the data node, the fourth shard set and the load value according to the data node, the fourth shard set and the corresponding third load value.

[0153] It should be noted that when it is necessary to obtain the load value corresponding to each set on the transfer node, the load value corresponding to each set on the transfer node can also be calculated in real time based on the above method for calculating the third load value.

[0154] Step 602: Detect whether there is a third shard set corresponding to the first preset tag in the second shard set.

[0155] The first preset tag is a tag indicating a high read / write frequency, and is a tag other than the third preset tag. The first preset tag may be one or more.

[0156] In this step, the read / write heat label of each second shard set is obtained, and the read / write heat label of each second shard set is compared with the first preset label. When the read / write heat label of the second shard set is the first preset label, the second shard set is determined as the third shard set.

[0157] Step 603, when there is a third shard set corresponding to the first preset label, determine the second average load value corresponding to the first preset label according to the first preset label, and the correspondence between the read-write heat label and the average load value, and detect whether there is a first shard set that meets the high read-write operating conditions in the third shard set according to the second load value and the second average load value corresponding to the third shard set.

[0158] When the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read / write operation condition.

[0159] In this step, when there is a third shard set corresponding to the first preset label, the load value corresponding to each third shard set is obtained, and the second average load value corresponding to the first preset label is determined according to the first preset label and the correspondence between the read-write heat label and the average load value. When the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read-write operating conditions.

[0160] It should be noted that, when determining the second average load value corresponding to the first preset tag, it is also possible to Figure 5 The method calculates in real time a second average load value corresponding to a second preset tag.

[0161] Based on the above embodiments, the present application provides a data balancing method, such as Figure 7 As shown, the specific steps include:

[0162] Step 701, for each data node, set a read-write heat label for each shard on the data node.

[0163] This step is Figure 4 The steps involved in the corresponding embodiments are the same and will not be described in detail here.

[0164] Step 702: According to each data node and the shard set included therein, a corresponding relationship between the data node, the shard set and the load value is established and stored.

[0165] This step is the same as establishing the correspondence between data nodes, shard sets and load values ​​in step 601, and will not be repeated here.

[0166] Step 703: Establish a corresponding relationship between the read / write heat tag and the average load value.

[0167] This step and Figure 5 The steps involved in the corresponding embodiments are the same and will not be described in detail here.

[0168] It should be noted that the execution order of step 701, step 702 and step 703 is not fixed. Step 701, step 702 and step 703 can be executed in sequence, or in other orders, and the embodiments of the present application are not limited thereto.

[0169] Step 704: Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first number of shards.

[0170] This step is the same as step 102 and will not be described in detail here.

[0171] Step 705: Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node.

[0172] This step is the same as step 102 and will not be described in detail here.

[0173] Step 706: When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

[0174] This step is the same as step 204 and will not be described in detail here.

[0175] Step 707: Obtain a second load value corresponding to each second shard set on the transfer node.

[0176] This step is the same as step 601 and will not be described in detail here.

[0177] Step 708: Detect whether there is a third shard set corresponding to the first preset tag in the second shard set.

[0178] This step is the same as step 602 and will not be described in detail here.

[0179] Step 709: When there is a third shard set corresponding to the first preset tag, in the third shard set, it is detected whether there is a first shard set that meets the high read / write operation condition.

[0180] This step is the same as step 603 and will not be described in detail here.

[0181] Step 710: When there is a first shard set that meets the high read / write operation condition, a data transfer task is generated according to the first shard set and the target node.

[0182] In this step, a shard is randomly selected from the first shard set, and a data transfer task is generated based on the shard and the target node.

[0183] Step 711, when there is no first shard set that meets the high read / write operation condition, determine a target node that meets the low load operation condition and meets the low load operation condition under a third preset label.

[0184] In this step, when there is no first shard set that meets the high read and write operating conditions, it means that the data node has no accumulated shards with high read and write frequencies, and there is no need to process the shards with high read and write frequencies. However, since the data node is in high-load operation, the accumulation of shards corresponding to the third preset label will also cause the data node to be in high-load operating conditions. Therefore, it is also necessary to determine the low-load operating conditions and the target node that meets the low-load operating conditions under the third preset label.

[0185] Step 712: Generate a data transfer task according to the other shard sets and the target node except the third shard set.

[0186] In this step, a shard is randomly selected from the set of other shards, and a data transfer task is generated based on the shard and the target node.

[0187] Step 713: Update the data in the cluster according to the data transfer task and continue to generate tasks.

[0188] In this step, according to the shard transfer result in the data transfer task, the shard corresponding to the node in the centralization is updated, and steps 701 to 703 are re-executed based on the updated shard. After that, tasks are continuously generated until there are no data nodes in the cluster that meet the low-load operation conditions.

[0189] Step 714: When no more data transfer tasks are generated, data transfer is performed based on the generated data transfer tasks.

[0190] In this step, when no data transfer task is generated, that is, when there are no data nodes in the cluster that meet the low-load operation condition, data transfer is performed based on the generated data transfer task.

[0191] like Figure 8 As shown, an embodiment of the present application provides a data equalization device, which corresponds to the method embodiment and specifically includes:

[0192] An acquisition unit 801 is used to acquire a performance indicator corresponding to each data node in the cluster and the number of first shards of shards included in each data node;

[0193] A first determining unit 802 is used to determine a transfer node that meets the high-load operation condition according to the performance indicator corresponding to each data node and the number of first shards;

[0194] A detection unit 803 is used to detect whether there is a first set of shards that meet the high read / write operation condition in the shards included in the transfer node, each shard in the first set of shards corresponding to a first preset label;

[0195] A second determining unit 804 is configured to determine, when there is a first shard set that meets the high read / write operation condition, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label according to the first preset label;

[0196] The generating unit 805 is used to generate a data transfer task according to the first shard set and the target node, so as to transfer the shards in the first shard set to the target node.

[0197] Optionally, the first determining unit 802 is configured to:

[0198] For each data node, obtain a first parameter value corresponding to the data node, and determine a first value corresponding to the data node according to the first number of shards corresponding to the data node and the first parameter value, where the first parameter value indicates a capability of the data node to store data;

[0199] Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value;

[0200] Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node;

[0201] When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

[0202] Optionally, the first determining unit 802 is configured to:

[0203] For each data node, a performance indicator corresponding to the data node is multiplied by an indicator weight to obtain a second value corresponding to the performance indicator, and a sum of the second value and the first value is determined as a first load value of the data node.

[0204] Optionally, the detection unit 803 is used to:

[0205] Obtain a second load value corresponding to each second shard set on the transfer node, wherein the read and write heat labels corresponding to each shard in the second shard set are the same;

[0206] In the second shard set, detecting whether there is a third shard set corresponding to a first preset tag, where the first preset tag is a tag indicating a high read / write frequency;

[0207] When there is a third shard set corresponding to the first preset label, the second average load value corresponding to the first preset label is determined according to the first preset label and the correspondence between the read-write heat label and the average load value; when the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read-write operating conditions.

[0208] Optionally, the apparatus further includes a first establishing unit 806, wherein the first establishing unit 806 is configured to:

[0209] For each data node, obtain the read-write heat label of each shard on the data node, divide the shards corresponding to the same read-write heat label into the same set to obtain at least one fourth shard set, obtain the number of second shards corresponding to each fourth shard set, determine the third load value corresponding to each fourth shard set based on the number of second shards corresponding to each fourth shard set and the first parameter value of the data node, and establish a correspondence between the data node, the shard set and the load value based on the data node, the fourth shard set on the data node and the third load value corresponding to each fourth shard set.

[0210] Optionally, the apparatus further includes a second establishing unit 807, wherein the second establishing unit 807 is configured to:

[0211] For each read / write heat label, determine the total number of shards corresponding to the read / write heat label in the cluster, and obtain the total number corresponding to the read / write heat label;

[0212] Determine a third average load value corresponding to each read / write heat label according to the total number corresponding to each read / write heat label and a second parameter corresponding to the cluster, wherein the second parameter indicates a data storage capability of the cluster;

[0213] According to each read / write heat label and the corresponding third average load value, a corresponding relationship between the read / write heat label and the average load value is established.

[0214] Optionally, the device further includes a marking unit 808, and the marking unit 808 is used to:

[0215] Determine the target business to which the slice to be marked belongs;

[0216] According to the target hotspot detection rule corresponding to the target business, detect whether the shard to be marked is a hotspot shard;

[0217] When the shard to be marked is a hotspot shard, obtaining a second preset tag corresponding to the target service, and setting the second preset tag as a read-write heat tag of the shard to be marked;

[0218] When the shard to be marked is a non-hotspot shard, a preset third preset tag is obtained, and the third preset tag is set as the read-write heat tag of the shard to be marked.

[0219] like Fig. 9 As shown, the embodiment of the present application provides a data balancing device, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0220] Memory 903, used for storing computer programs;

[0221] In one embodiment of the present application, the processor 901 is used to implement the data balancing method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 903, including:

[0222] Get the performance indicators corresponding to each data node in the cluster and the number of first shards of the included shards;

[0223] Determine a transfer node that meets the high-load operation condition according to the performance index corresponding to each data node and the number of first shards;

[0224] Among the shards included in the transfer node, detecting whether there is a first shard set that meets the high read / write operation condition, each shard in the first shard set corresponding to a first preset label;

[0225] When there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label;

[0226] A data transfer task is generated according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

[0227] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps performed by the data balancing method provided in any of the aforementioned method embodiments are implemented.

[0228] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0230] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0231] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data balancing method, characterized in that: The method comprises: Obtain the performance indicators corresponding to each data node in the cluster and the number of first shards of the shards included in each data node; Determine a transfer node that meets the high-load operation condition according to the performance index corresponding to each data node and the number of first shards; Among the shards included in the transfer node, detecting whether there is a first shard set that meets the high read / write operation condition, each shard in the first shard set corresponding to a first preset label; When there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label; A data transfer task is generated according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

2. The method according to claim 1, characterized in that: The step of determining a transfer node that meets the high-load operation condition according to the performance indicator corresponding to each data node and the number of first shards includes: For each data node, obtain a first parameter value corresponding to the data node, and determine a first value corresponding to the data node according to the first number of shards corresponding to the data node and the first parameter value, where the first parameter value indicates a capability of the data node to store data; Determine a first load value corresponding to each data node according to the performance indicator corresponding to each data node and the first value; Determine a first average load value corresponding to the cluster according to the first load value corresponding to each data node; When there is a data node corresponding to the first load value greater than the first average load value, the data node is determined as a transfer node that meets the high-load operation condition.

3. The method according to claim 2, characterized in that: The determining, according to the performance indicator and the first value corresponding to each data node, a first load value corresponding to each data node includes: For each data node, a performance indicator corresponding to the data node is multiplied by an indicator weight to obtain a second value corresponding to the performance indicator, and a sum of the second value and the first value is determined as a first load value of the data node.

4. The method according to claim 1, characterized in that: The detecting whether there is a first set of shards meeting the high read / write operation condition in the shards included in the transfer node includes: Obtain a second load value corresponding to each second shard set on the transfer node, wherein the read and write heat labels corresponding to each shard in the second shard set are the same; In the second shard set, detecting whether there is a third shard set corresponding to a first preset tag, where the first preset tag is a tag indicating a high read / write frequency; When there is a third shard set corresponding to the first preset label, the second average load value corresponding to the first preset label is determined according to the first preset label and the correspondence between the read-write heat label and the average load value; when the second load value corresponding to the third shard set is greater than the second average load value, the third shard set is determined as the first shard set that meets the high read-write operating conditions.

5. The method according to claim 1, characterized in that: The method further comprises: For each data node, obtain the read-write heat label of each shard on the data node, divide the shards corresponding to the same read-write heat label into the same set to obtain at least one fourth shard set, obtain the number of second shards corresponding to each fourth shard set, determine the third load value corresponding to each fourth shard set based on the number of second shards corresponding to each fourth shard set and the first parameter value of the data node, and establish a correspondence between the data node, the shard set and the load value based on the data node, the fourth shard set on the data node and the third load value corresponding to each fourth shard set.

6. The method according to claim 1, characterized in that: The method further comprises: For each read / write heat label, determine the total number of shards corresponding to the read / write heat label in the cluster, and obtain the total number corresponding to the read / write heat label; Determine a third average load value corresponding to each read / write heat label according to the total number corresponding to each read / write heat label and a second parameter corresponding to the cluster, wherein the second parameter indicates a data storage capability of the cluster; According to each read / write heat label and the corresponding third average load value, a corresponding relationship between the read / write heat label and the average load value is established.

7. The method according to claim 5, characterized in that: The method further comprises: Determine the target business to which the slice to be marked belongs; According to the target hotspot detection rule corresponding to the target business, detect whether the shard to be marked is a hotspot shard; When the shard to be marked is a hotspot shard, obtaining a second preset tag corresponding to the target service, and setting the second preset tag as a read-write heat tag of the shard to be marked; When the shard to be marked is a non-hotspot shard, a preset third preset tag is obtained, and the third preset tag is set as the read-write heat tag of the shard to be marked.

8. A data equalization device, characterized in that: The device comprises: An acquisition unit, used to acquire a performance indicator corresponding to each data node in the cluster and the number of first shards of shards included in each data node; A first determining unit, configured to determine a transfer node that meets a high-load operation condition according to a performance indicator corresponding to each data node and the number of first shards; A detection unit, configured to detect whether there is a first set of shards that meet the high read / write operation condition in the shards included in the transfer node, each shard in the first set of shards corresponding to a first preset label; A second determination unit is configured to determine, when there is a first shard set that meets the high read / write operation condition, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label according to the first preset label; A generating unit is used to generate a data transfer task according to the first shard set and the target node, so as to transfer the shards in the first shard set to the target node.

9. A data equalization device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: Obtain the performance indicators corresponding to each data node in the cluster and the number of first shards of the shards included in each data node; Determine a transfer node that meets the high-load operation condition according to the performance index corresponding to each data node and the number of first shards; Among the shards included in the transfer node, detecting whether there is a first shard set that meets the high read / write operation condition, each shard in the first shard set corresponding to a first preset label; When there is a first shard set that meets the high read / write operation condition, determine, according to the first preset label, a target node that meets the low load operation condition and meets the low read / write operation condition under the first preset label; A data transfer task is generated according to the first shard set and the target node to transfer the shards in the first shard set to the target node.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data equalization method according to any one of claims 1 to 7 is implemented.

Images