Data self-balancing method and device and related equipment

Abstract

The invention relates to the technical field of data processing, in particular to a data self-balancing method and device and related equipment. The method is applied to scheduling decision-making equipment and comprises the steps that key index information of all nodes included in a Kafka cluster is obtained, and the key index of one node is used for representing the load state of the node; respectively calculating the node health degree of each node based on the key index information of each node; when it is determined that the health degree of any node does not meet the preset requirement, a data balance scheme is generated based on the number of partitions of the subject borne by each node and the flow information of each subject partition, and the subject borne by each node and / or the partition of the borne subject are adjusted based on the data balance scheme.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data self-balancing method, device and related equipment. Background Art

[0002] Apache Kafka (hereinafter referred to as Kafka) is an open source distributed event streaming platform that is widely used in high-performance data pipelines, stream analysis, data integration, and mission-critical applications. The main features of Kafka are: distribution, persistence, elasticity, reliability, high performance, high reliability, timeliness, and scalability. The main application scenarios are: publish / subscribe, event bus, storage, and complex transaction processing. Among them, publish / subscribe is the most basic usage mode of Kafka. It can deliver messages to a specified consumer cluster and allow consumers to process messages asynchronously when receiving them. Kafka also provides an event bus function that can share messages between subscribers and track the processing process in real time. It is also one of the important tools for implementing microservices. Kafka can also be used to build a large-scale persistent event system that can store events on different sources and deliver them according to different partitions. In addition, Kafka can also be applied to complex transaction processing scenarios, such as distributed batch computing, streaming computing, and visibility and atomic transactions.

[0003] In a Kafka cluster, traffic changes, topic creation and deletion, and broker demise and startup occur all the time, and these changes may cause traffic to be unevenly distributed among the nodes in the cluster, resulting in resource waste and affecting business stability. At this time, it is necessary to actively move different partitions of topics between nodes to achieve the purpose of balancing traffic and data.

[0004] So, how to automatically balance the data of each node in the Kafka cluster is a problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a data self-balancing method, device and related equipment.

[0006] In a first aspect, the present application provides a data self-balancing method, which is applied to a scheduling decision device, and the method includes:

[0007] Get the key indicator information of each node in the Kafka cluster, where the key indicator of a node is used to characterize the load status of the node;

[0008] Based on the key indicator information of each node, calculate the node health of each node;

[0009] When it is determined that the health of any node does not meet the preset requirements, a data balancing plan is generated based on the number of partitions of the topics carried by each node and the traffic information of each topic partition, and the topics carried by each node and / or the partitions of the topics carried are adjusted based on the data balancing plan.

[0010] Optionally, a collection tool is deployed on each node included in the Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic; the step of obtaining the key indicator information of each node included in the Kafka cluster includes:

[0011] Receive key indicator information of each node sent by the target topic.

[0012] Optionally, the key indicator information of a node includes: traffic information of the topic partition carried by the node, and resource utilization information of the node.

[0013] Optionally, the collection tool deployed on each node collects key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic, and the scheduling decision device receives the key indicator information of each node sent by the target topic;

[0014] Based on the key indicator information of each node, the steps of calculating the node health of each node include:

[0015] The scheduling decision device is based on the second period T2. When determining the end of the current period, the scheduling decision device calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein T2=n multiplied by T1, n is a positive integer greater than or equal to 2.

[0016] In a second aspect, the present application provides a data self-balancing device, which is applied to a scheduling decision device, and the device includes:

[0017] An acquisition unit, used to acquire key indicator information of each node included in the Kafka cluster, wherein the key indicator of a node is used to characterize the load status of the node;

[0018] A calculation unit, used to calculate the node health of each node based on the key indicator information of each node;

[0019] The execution unit is used to generate a data balancing plan based on the number of partitions of the topics carried by each node and the traffic information of each topic partition when it is determined that the health of any node does not meet the preset requirements, and adjust the topics carried by each node and / or the partitions of the topics carried based on the data balancing plan.

[0020] Optionally, a collection tool is deployed on each node included in the Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic; when obtaining the key indicator information of each node included in the Kafka cluster, the acquisition unit is specifically used to:

[0021] Receive key indicator information of each node sent by the target topic.

[0022] Optionally, the key indicator information of a node includes: traffic information of the topic partition carried by the node, and resource utilization information of the node.

[0023] Optionally, the collection tool deployed on each node collects key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic, and the scheduling decision device receives the key indicator information of each node sent by the target topic;

[0024] When calculating the node health of each node based on the key indicator information of each node, the calculation unit is specifically used to:

[0025] The scheduling decision device is based on the second period T2. When determining the end of the current period, the scheduling decision device calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein T2=n multiplied by T1, n is a positive integer greater than or equal to 2.

[0026] In a third aspect, an embodiment of the present application provides a data self-balancing device, the data self-balancing device comprising:

[0027] A memory for storing program instructions;

[0028] The processor is used to call the program instructions stored in the memory, and execute the steps of the method as described in any one of the first aspects above according to the obtained program instructions.

[0029] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the steps of the method described in any one of the first aspects above.

[0030] In summary, the data self-balancing method provided in the embodiment of the present application is applied to a scheduling decision-making device, and the method includes: obtaining key indicator information of each node included in the Kafka cluster, wherein the key indicator of a node is used to characterize the load status of the node; based on the key indicator information of each node, respectively calculating the node health of each node; when it is determined that the health of any node does not meet the preset requirements, based on the number of partitions of the topic carried by each node and the traffic information of each topic partition, generating a data balancing plan, and adjusting the topics carried by each node and / or the partitions of the carried topics based on the data balancing plan.

[0031] The data self-balancing method provided in the embodiment of the present application is adopted, through the lightweight data automatic balancing third-party device / integration and the functional components in the Kafka cluster, the cluster node status is continuously monitored, and the migration of the topic partitions carried by each node is automatically performed. The risk of manually performing data rebalancing is greatly reduced, and the service interruption time is shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0033] Figure 1 This is a schematic diagram of the current data migration process;

[0034] Figure 2 A detailed flow chart of a data self-balancing method provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a data self-balancing device provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the hardware architecture of a data self-balancing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0038] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0039] Currently, Kafka clusters do not automatically assign any data partitions. Therefore, typically when adding machines (nodes) to the cluster or when data skew occurs, users want to migrate some existing data to suitable nodes. The current process of migrating data is manually initiated, and replication between data is fully automated. During the current data migration process, the Kafka cluster adds the new server as a follower of the partition it is migrating and allows it to fully replicate the existing data in that partition. When the new server fully replicates the contents of the partition and joins the in-sync replica, one of the existing replicas will delete the data for its partition. The partition reassignment tool can be used to move partitions across brokers. An ideal partition distribution will ensure that all brokers have an even data load and partition size. The partition reassignment tool cannot automatically study the data distribution in the Kafka cluster and move partitions to achieve an even load distribution. Therefore, administrators must figure out which topics or partitions should be moved. That is, see Figure 1 As shown in the figure, it is a schematic diagram of the current data migration process; the operation and maintenance personnel observe the data distribution of the Kafka cluster and obtain the observation results; the operation and maintenance personnel then generate a migration plan based on the observation results, and send the migration plan to the Kafka cluster to execute the plan.

[0040] However, if the scale of online Kafka nodes is large, manually planning and executing migration data balancing is very time-consuming and labor-intensive. And because the rebalancing task is performed centrally, the resource preemption caused by data synchronization will greatly affect Kafka's service capabilities.

[0041] This application aims at the shortcomings of the current Kafka cluster data rebalancing and proposes an optimization solution for automatic rebalancing of Kafka cluster data.

[0042] For example, see Figure 2 As shown, it is a detailed flow chart of a data self-balancing method provided in an embodiment of the present application. The method is applied to a scheduling decision device, and the method includes the following steps:

[0043] Step 100: Obtain key indicator information of each node included in the Kafka cluster, wherein the key indicator of a node is used to characterize the load status of the node.

[0044] In the embodiment of the present application, a collection tool is deployed on each node included in the above Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic. Then, when obtaining the key indicator information of each node included in the Kafka cluster, a better implementation method is: receiving the key indicator information of each node sent by the above target topic.

[0045] Specifically, the collection tool deployed on each node collects the key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the above-mentioned target topic, and the above-mentioned scheduling decision device receives the key indicator information of each node sent by the above-mentioned target topic.

[0046] In one implementation of the embodiment of the present application, the scheduling decision device is any node in the Kafka cluster (such as the master node), and the scheduling decision device includes a collection function module (collection tool) and a scheduling decision function module, and the collection function module is deployed on each node of the Kafka cluster, and the scheduling decision function module is deployed on the master node. In another implementation of the present application, the scheduling decision device can also be a third-party device, and the scheduling decision device includes a collection function module and a scheduling decision function module, and the collection function modules of the scheduling decision device are respectively deployed on each node of the Kafka cluster.

[0047] Next, a topic (such as the target topic) is specified from the topics (Topic) included in the Kafka cluster, and the scheduling decision function module subscribes to the target topic. Then, the collection tool deployed on each node collects the key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic. Since the scheduling decision function module subscribes to the target topic, the target topic will actively send the newly collected information to the subscriber, and the scheduling decision function module will receive the key indicator information of each node sent by the target topic.

[0048] In the embodiment of the present application, the so-called key indicator information of the node refers to: network import and export traffic (Topic-Partition traffic), CPU, disk, memory utilization, etc. That is, the hardware resource utilization rate of each node, the partition traffic size of each topic carried, etc.

[0049] Step 110: Based on the key indicator information of each node, the node health of each node is calculated respectively.

[0050] In the embodiment of the present application, the step of calculating the node health of each node based on the key indicator information of each node includes:

[0051] The above-mentioned scheduling decision device is based on the second period T2. When determining the end of the current period, it calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein the above T2 = n multiplied by T1, n is a positive integer greater than or equal to 2.

[0052] For example, the first cycle T1 is 1 minute, and the second cycle T2 is 5 minutes; each T2 cycle includes 5 T1 cycles, and for each type of indicator, includes 5 specific values ​​of indicators. In the embodiment of the present application, the average value of each type of indicator in each second cycle can be calculated as the indicator value of this type of indicator in the second cycle. For example, for the CPU usage of a node, at the end of the current second cycle, the 5 CPU usage values ​​included in the current second cycle are averaged to obtain the CPU usage value of the node in the current second cycle.

[0053] In an embodiment of the present application, the weight values ​​of various indicators can be configured in advance according to needs. Then, after calculating the indicator values ​​of various indicators in the current second period, the node health of each node can be calculated by weighted summation.

[0054] In actual applications, if the CPU, disk, and memory usage of a node is higher, the corresponding node health is lower. Similarly, if the CPU, disk, and memory usage of a node is lower, the corresponding node health is higher. If the network import and export traffic of the topic partition carried by a node is larger, the corresponding node health is lower. Similarly, if the network import and export traffic of the topic partition carried by a node is smaller, the corresponding node health is higher.

[0055] That is, the node health of a node is inversely proportional to the hardware resource utilization of the node and the network import and export traffic of the topic partition carried by the node.

[0056] Step 120: When it is determined that the health of any node does not meet the preset requirements, a data balancing plan is generated based on the number of partitions of the topics carried by each node and the traffic information of each topic partition, and the topics carried by each node and / or the partitions of the topics carried are adjusted based on the above data balancing plan.

[0057] Specifically, a health threshold is preset, and when it is detected that the health of any node is less than the health threshold, it is determined that the subject partition data of the node needs to be adjusted.

[0058] For example, part of the topic partition data of a node whose health threshold does not meet the preset requirements is migrated to other nodes with higher health.

[0059] In the embodiment of the present application, the node whose health does not meet the preset requirements is defined as the target node. Then, when generating a data balancing plan based on the number of partitions of the topics carried by each node and the network traffic information of each topic partition, any of the following data balancing methods can be used: If it is determined that a topic has multiple partitions at the target node whose health does not meet the preset requirements, one or more partitions of the topic are migrated to other nodes so that multiple partitions of a topic are carried on different nodes; if it is determined that the network traffic of a topic partition is much larger than the network traffic of other topic partitions, the topic partition data can be migrated to other nodes, .... Until the next node health calculation cycle is over, when it is determined that the health of each node meets the preset requirements, the data self-balancing of the Kafka node is completed.

[0060] Based on the same inventive concept as the above-mentioned embodiment of the invention, for example, refer to Figure 3 FIG. 1 is a schematic diagram of a data self-balancing device provided in an embodiment of the present application. The device is applied to a scheduling decision device. The device includes:

[0061] An acquisition unit 30 is used to acquire key indicator information of each node included in the Kafka cluster, wherein the key indicator of a node is used to characterize the load status of the node;

[0062] A calculation unit 31, used to calculate the node health of each node based on the key indicator information of each node;

[0063] The execution unit 32 is used to generate a data balancing plan based on the number of partitions of the topics carried by each node and the traffic information of each topic partition when it is determined that the health of any node does not meet the preset requirements, and adjust the topics carried by each node and / or the partitions of the topics carried based on the data balancing plan.

[0064] Optionally, a collection tool is deployed on each node included in the Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic; when obtaining the key indicator information of each node included in the Kafka cluster, the acquisition unit 30 is specifically used to:

[0065] Receive key indicator information of each node sent by the target topic.

[0066] Optionally, the key indicator information of a node includes: traffic information of the topic partition carried by the node, and resource utilization information of the node.

[0067] Optionally, the collection tool deployed on each node collects key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic, and the scheduling decision device receives the key indicator information of each node sent by the target topic;

[0068] When calculating the node health of each node based on the key indicator information of each node, the calculation unit 31 is specifically used to:

[0069] The scheduling decision device is based on the second period T2. When determining the end of the current period, the scheduling decision device calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein T2=n multiplied by T1, n is a positive integer greater than or equal to 2.

[0070] The above units may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a certain unit above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program codes. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0071] Furthermore, the data self-balancing device provided in the embodiment of the present application, from the hardware level, the hardware architecture diagram of the data self-balancing device can be seen in Figure 4 As shown, the data self-balancing device may include: a memory 40 and a processor 41,

[0072] The memory 40 is used to store program instructions; the processor 41 calls the program instructions stored in the memory 40 and executes the above method embodiment according to the obtained program instructions. The specific implementation method and technical effect are similar and will not be repeated here.

[0073] Optionally, the present application also provides a data self-balancing device, comprising at least one processing element (or chip) for executing the above method embodiment.

[0074] Optionally, the present application also provides a program product, such as a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the above method embodiments.

[0075] Here, the machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0076] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0077] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data self-balancing method, characterized in that: Applied to a scheduling decision device, the method comprises: Get the key indicator information of each node in the Kafka cluster, where the key indicator of a node is used to characterize the load status of the node; Based on the key indicator information of each node, calculate the node health of each node; When it is determined that the health of any node does not meet the preset requirements, a data balancing plan is generated based on the number of partitions of the topics carried by each node and the traffic information of each topic partition, and the topics carried by each node and / or the partitions of the topics carried are adjusted based on the data balancing plan.

2. The method according to claim 1, characterized in that A collection tool is deployed on each node included in the Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic; The steps to obtain key indicator information of each node in the Kafka cluster include: Receive key indicator information of each node sent by the target topic.

3. The method according to claim 1 or 2, characterized in that The key indicator information of a node includes: traffic information of the topic partitions carried by the node and resource utilization information of the node.

4. The method according to claim 1 or 2, characterized in that: The collection tool deployed on each node collects the key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic, and the scheduling decision device receives the key indicator information of each node sent by the target topic; Based on the key indicator information of each node, the steps of calculating the node health of each node include: The scheduling decision device is based on the second period T2. When determining the end of the current period, the scheduling decision device calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein T2=n multiplied by T1, n is a positive integer greater than or equal to 2.

5. A data self-balancing device, characterized in that: Applied to a scheduling decision device, the device comprises: An acquisition unit, used to acquire key indicator information of each node included in the Kafka cluster, wherein the key indicator of a node is used to characterize the load status of the node; A calculation unit, used to calculate the node health of each node based on the key indicator information of each node; The execution unit is used to generate a data balancing plan based on the number of partitions of the topics carried by each node and the traffic information of each topic partition when it is determined that the health of any node does not meet the preset requirements, and adjust the topics carried by each node and / or the partitions of the topics carried based on the data balancing plan.

6. The device according to claim 5, characterized in that A collection tool is deployed on each node included in the Kafka cluster, and the collection tool of each node sends the collected key indicator information to the target topic; when obtaining the key indicator information of each node included in the Kafka cluster, the acquisition unit is specifically used to: Receive key indicator information of each node sent by the target topic.

7. The device according to claim 5 or 6, characterized in that The key indicator information of a node includes: traffic information of the topic partitions carried by the node and resource utilization information of the node.

8. The device according to claim 5 or 6, characterized in that The collection tool deployed on each node collects the key indicator information of the node based on the first period T1, and sends the collected key indicator information of the node to the target topic, and the scheduling decision device receives the key indicator information of each node sent by the target topic; When calculating the node health of each node based on the key indicator information of each node, the calculation unit is specifically used to: The scheduling decision device is based on the second period T2. When determining the end of the current period, the scheduling decision device calculates the average value of each key indicator information of each node according to multiple key indicator information of each node received in the current period, and for each node, according to the average value of each key indicator of the node and the weight value set in advance for each key indicator, a weighted summation method is used to calculate the node health of the node, wherein T2=n multiplied by T1, n is a positive integer greater than or equal to 2.

9. A data self-balancing device, characterized in that: The data self-balancing device comprises: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps of the method according to any one of claims 1 to 4 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the steps of the method according to any one of claims 1 to 4.

Images