System and method for usage-based leader balancing of topic partitions in kafka cluster

Abstract

The present invention relates to a system and a method for usage-based leader balancing of topic partitions in a Kafka cluster and, more specifically, to a system and a method for usage-based leader balancing of topic partitions in a Kafka cluster, the system and method providing a new partition leader balancing method considering network usage rather than a simple criterion of evenly distributing the number of leader partitions.

Description

A system and method for leader balancing of usage-based topic partitions in a Kafka cluster.

[0001] The present invention relates to a system and method for leader balancing of a usage-based topic partition in a Kafka cluster, and more particularly, to a system and method for leader balancing of a usage-based topic partition in a Kafka cluster for providing a new partition leader balancing method that takes network usage into account rather than a criterion of simply evenly distributing the number of leader partitions.

[0002] Apache Kafka is a highly scalable, high-throughput distributed messaging system that guarantees reliable and continuous message delivery. Kafka applies the producer-consumer problem and uses three system components—Producer, Consumer, and Broker—to manage large volumes of data. It partitions topics and stores them sequentially. This stored data is then sequentially delivered to consumers for efficient processing.

[0003] The number of partitions and leader partitions is the criterion for evenly distributing the partitions and leader partitions based on topics on each Kafka broker server in the Kafka cluster, and the reason for distributing them is to evenly distribute network usage.

[0004] However, since the amount of messages written to and read from each topic in a Kafka cluster is different and the network usage is also different, there is a limitation that network usage cannot be evenly distributed simply by evenly distributing the number of leader partitions.

[0005] Accordingly, in the relevant technical field, there is a need for technological development to provide a new partition leader balancing method that takes network usage into account, rather than simply distributing the number of leader partitions evenly.

[0006] [Prior Art Literature]

[0007] (Patent Document 1) Republic of Korea Patent Application No. 10-2017-0027318 (2017.03.02) "System and Method for Large Scale Image Processing in Real-Time Environments Using Apache Kafka"

[0008] (Patent Document 2) Republic of Korea Patent Application No. 10-2018-0136731 (November 8, 2018) "System and method for transmission of data in real time using multiple KAFKA"

[0009] The present invention is intended to solve the above problems, and to provide a new partition leader balancing method that takes network usage into account rather than a standard that simply evenly distributes the number of leader partitions, thereby providing a usage-based topic partition leader balancing system and method in a Kafka cluster.

[0010] In addition, the present invention provides a system and method for leader balancing of usage-based topic partitions in a Kafka cluster by providing an information category structure to provide a new partition leader balancing method that takes network usage into account rather than a criterion of simply evenly distributing the number of leader partitions.

[0011] The purposes of the present invention are not limited to the purposes mentioned above, and other purposes not mentioned will be clearly understood by those skilled in the art from the description below.

[0012] In order to achieve the above purpose, the leader balancing system of usage-based topic partition in a Kafka cluster according to an embodiment of the present invention is

[0013] Multiple Kafka broker servers (100); and

[0014] By traversing the broker list of the above Kafka broker server (100), the Kafka broker server (100) with the maximum / minimum value of the sum of the network traffic estimates of each Kafka broker server (100) {sum(Assumed Traffic)} is found and the difference is checked, and SATR DIFF If it is below the set threshold, the adjustment operation is stopped and Bid MAX Select a leader partition (Leader) that meets the selection conditions from the leader partition list (Leaders) of the Kafka broker server (100), and select the selected partition (P SEL ) to Bid MIN It is characterized by including a leader balance adjustment module (200) that moves to a Kafka broker server (100).

[0015] At this time, the leader balance control module (200)

[0016] A leader balancing system for usage-based topic partitions in a Kafka cluster can be provided, characterized by performing abort operations if no partitions have been moved.

[0017] In addition, the leader balance control module (200)

[0018] A leader balancing system for usage-based topic partitions in a Kafka cluster can be provided, characterized by repeating a process for a set maximum number of iterations.

[0019] In order to achieve the above purpose, a leader balancing system for usage-based topic partitions in a Kafka cluster according to another embodiment of the present invention is provided.

[0020] Multiple Kafka broker servers (100); and

[0021] By traversing the broker list of the above Kafka broker server (100), the Kafka broker server (100) that is the broker having the maximum / minimum network traffic estimate (Assumed Traffic) for each leader partition (Leader) that constitutes the leader partition list (Leaders) is found and the difference value is checked, and LDR DIFF If it is below the set threshold, the adjustment operation is stopped and Bid MAX Select a leader partition that meets the selection conditions from the leader partition list (Leaders) of the corresponding Kafka broker server (100), and select the selected partition (P SEL ) to Bid MIN It is characterized by including a leader balance adjustment module (200) that moves to a Kafka broker server (100).

[0022] At this time, the leader balance control module (200)

[0023] A leader balancing system for usage-based topic partitions in a Kafka cluster can be provided, characterized by performing abort operations if no partitions have been moved.

[0024] In addition, the leader balance control module (200)

[0025] A leader balancing system for usage-based topic partitions in a Kafka cluster can be provided, characterized by performing repetitions for a set maximum number of repetitions.

[0026] In order to achieve the above purpose, a method for adjusting leader balance of usage-based topic partitions in a Kafka cluster according to an embodiment of the present invention is provided.

[0027] The first step is to traverse the broker list of Kafka broker servers (100) to find the Kafka broker server (100) having the maximum / minimum value of the sum {sum(Assumed Traffic)} of the network traffic estimates of each Kafka broker server (100) and check the difference;

[0028] SATR DIFF The second step is to stop the adjustment operation if it is below the set threshold;

[0029] Bid MAX A third step of selecting a leader partition (Leader) that meets the selection conditions from the leader partition list (Leaders) of the Kafka broker server (100); and

[0030] Selected partition (P SEL ) to Bid MIN It is characterized by including a fourth step of moving to a Kafka broker server (100);

[0031] At this time, after the above 4th step,

[0032] A method for leader balancing of usage-based topic partitions in a Kafka cluster can be provided, characterized in that it further includes a fifth step of performing a job suspension if there is no moved partition;

[0033] Also, after the above 5th step,

[0034] A method for leader balancing of a usage-based topic partition in a Kafka cluster can be provided, characterized in that the entire process of steps 1 to 5 is repeated a set maximum number of repetitions.

[0035] In order to achieve the above purpose, a method for leader balancing of usage-based topic partitions in a Kafka cluster according to another embodiment of the present invention is provided.

[0036] A first step of traversing the broker list of the Kafka broker server (100) to find the Kafka broker server (100) that has the maximum / minimum network traffic estimate (Assumed Traffic) for each leader partition (Leader) that constitutes the leader partition list (Leaders) and checking the difference value;

[0037] LDR DIFF A second step is to perform a stop operation if the value is below the set threshold;

[0038] Bid MAX A third step of selecting a leader partition that meets the selection conditions from the leader partition list (Leaders) of the corresponding Kafka broker server (100); and

[0039] Selected partition (P SEL ) to Bid MIN It is characterized by including a fourth step of moving to a Kafka broker server (100);

[0040] At this time, after the above 4th step,

[0041] A method for leader balancing of usage-based topic partitions in a Kafka cluster can be provided, characterized in that it further includes a fifth step of performing a job suspension if there is no moved partition;

[0042] Also, after the above 5th step,

[0043] A method for leader balancing of a usage-based topic partition in a Kafka cluster can be provided, characterized in that the entire process of steps 1 to 5 is repeated a set maximum number of repetitions.

[0044] The system and method for leader balancing of usage-based topic partitions in a Kafka cluster according to an embodiment of the present invention provides an effect of providing a new partition leader balancing method that takes network usage into account rather than a criterion of simply evenly distributing the number of leader partitions.

[0045] In addition, the system and method for leader balancing of usage-based topic partitions in a Kafka cluster according to another embodiment of the present invention provides an effect of providing an information category structure to provide a new partition leader balancing method that takes network usage into account rather than a criterion for simply evenly distributing the number of leader partitions.

[0046] Figure 1 is a diagram explaining the Kafka cluster structure.

[0047] FIG. 2 is a diagram for explaining the relationship between an existing Kafka broker server-topic-partition and a leader partition for configuring a leader balancing system (1) of a usage-based topic partition in a Kafka cluster according to an embodiment of the present invention.

[0048] FIG. 3 is a diagram showing a data structure provided by a leader balancing system (1) of a usage-based topic partition in a Kafka cluster according to an embodiment of the present invention.

[0049] FIG. 4 is a diagram showing the structure of a leader balancing system (1) for usage-based topic partitioning in a Kafka cluster according to an embodiment of the present invention.

[0050] FIG. 5 and FIG. 6 are diagrams illustrating a method for adjusting leader balance of usage-based topic partitions in a Kafka cluster according to an embodiment of the present invention.

[0051] Hereinafter, a detailed description of preferred embodiments of the present invention will be provided with reference to the attached drawings. In the following description of the present invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the present invention.

[0052] In this specification, when a component 'transmits' data or a signal to another component, it means that the component can transmit the data or signal directly to the other component, or can transmit the data or signal to the other component via at least one other component.

[0053] Figure 1 is a diagram explaining the Kafka cluster structure.

[0054] Referring to FIG. 1, a leader balancing system (1) of a topic partition in a Kafka cluster configures a Kafka cluster composed of multiple Kafka broker servers (100), and message data entering the Kafka cluster is stored in a logical structure called a topic within Kafka.

[0055] That is, a large amount of data can be stored in order by setting topics and configuring partitions based on the topics.

[0056] At this time, one topic can have one or more replicas, and each replica is stored in a different Kafka broker server (100), so that failure of a specific Kafka broker server (100) can be prepared.

[0057] Here, a topic is actually composed of one or more partitions that are stored on physical storage devices.

[0058] Therefore, when replicas are configured for a topic, physically identical partitions are stored in one or more Kafka broker servers (100).

[0059] One of the multiple replicas of this partition is treated as the original and is called the leader partition, while the rest are called follower partitions.

[0060] Messages entering the Kafka cluster are added to the end of the leader partition and are read by follower partitions and message consumers. Therefore, all write / read operations occur in the Kafka broker server (100) that has the leader partition.

[0061] A Kafka cluster can have multiple topics, and each topic can have multiple partitions, so a single Kafka cluster can have tens to tens of thousands of partitions.

[0062] FIG. 2 is a diagram for explaining the relationship between an existing Kafka broker server-topic-partition and a leader partition for configuring a leader balancing system (1) of a usage-based topic partition in a Kafka cluster according to an embodiment of the present invention.

[0063] Referring to Figure 2, when adding a topic while using a Kafka cluster, partitions can be distributed as in the example of Figure 2.

[0064] In the embodiment of FIG. 2, the number of brokers in the Kafka cluster, i.e., the number of Kafka broker servers (100), is 3, the number of topics is 3, A, B, and C, topic A is composed of 3 partitions / 2 replicas, topic B is composed of 4 partitions / 1 replica, and topic C can be composed of 2 partitions / 3 replicas, and the leader of each topic-partition is indicated by color.

[0065] In this configuration, most data input / output (IN / OUT) occurs in the color-coded leader partition.

[0066] Meanwhile, when using a Kafka cluster by adding and deleting topics on each Kafka broker server (100), an imbalance in the number of leader partitions on each Kafka broker server (100) may occur, and each topic-partition may have different data usage.

[0067] That is, to address the problem of these existing methods, the Kafka cluster is configured so that when creating a topic / partition, partitions are evenly distributed among each Kafka broker server (100), and leader partitions are also configured so that they are evenly distributed.

[0068] The criteria for evenly distributing partitions and leader partitions in a Kafka cluster is the number of partitions and leader partitions, and the reason for distributing them is to evenly distribute network usage.

[0069] However, since the amount of messages written to and read from each topic in a Kafka cluster is different, the network usage also varies accordingly.

[0070] Therefore, there is a limitation that network usage cannot be evenly distributed simply by evenly distributing the number of leader partitions. In the following, we will specifically examine a leader balance adjustment method for usage-based topic partitions in a Kafka cluster, focusing on a leader balance adjustment system (1) for usage-based topic partitions in a Kafka cluster according to the present invention.

[0071] FIG. 3 is a diagram illustrating a data structure provided by a leader balancing system (1) for usage-based topic partitioning in a Kafka cluster according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a structure of a leader balancing system (1) for usage-based topic partitioning in a Kafka cluster according to an embodiment of the present invention.

[0072] First, referring to FIG. 4, a leader balancing system (1) for usage-based topic partitioning in a Kafka cluster may have a structure in which a leader balancing module (200) is added to the existing structure of FIG. 1.

[0073] Next, referring to FIG. 3, a usage-based topic partition leader balancing system (1) in a Kafka cluster according to an embodiment of the present invention can provide a data structure for providing a new partition leader balancing method that considers network usage rather than a simple quantity criterion to solve existing problems. By adjusting the leader balance based on network usage through this data structure, the load can be evenly distributed across all Kafka broker servers (100) in the Kafka cluster.

[0074] First, as a first category structure, information on individual components (Broker1, Broker2, Broker3) of each Kafka broker server (100) corresponding to the list of all brokers (Brokers) of the Kafka cluster is structured.

[0075] Next, as a second category structure, it has information about ① broker ID (Broker.id) for each Kafka broker server (100) corresponding to an individual component of the first category structure, ② leader partition list (Leaders) of each Kafka broker server (100), and ③ sum of network traffic estimates {sum (Assumed Traffic)}.

[0076] More specifically, the broker ID (Broker.id) corresponds to the unique ID number of the Kafka broker server (100). The leader partition list (Leaders) represents the list of leader partitions currently owned by the Kafka broker server (100), and the sum of network traffic estimates {sum(Assumed Traffic)} means the sum of network traffic estimates of each leader partition owned by the Kafka broker server (100).

[0077] The following third category structure is information about each leader partition (Leader) of the leader partition list (Leaders) belonging to each Kafka broker server (100) of the second category structure, and has information about topic information (Topic), partition number (Partition Number), partition commit log size (single replica) (Logsize), and network traffic estimate (Assumed Traffic).

[0078] Partition commit log size (single replica) (Logsize) can be collected from each Kafka broker server (100), and network traffic estimate (Assumed Traffic) can be calculated by “Partition commit log size / Topic commit log size * Topic network traffic statistics”.

[0079] In the following 4th category structure, each leader partition (Leader) that constitutes the leader partition list (Leaders) of the 3rd category structure has topic information (Topic) of each partition corresponding to the topic information (Topic), including the topic name (Name), the topic's commit log size (single replica) (Logsize), and network traffic statistics (sum or average for a certain period) (Traffic).

[0080] Here, the topic's commit log size (single replica) (Logsize) is calculated by adding up the commit log sizes of all partitions belonging to the topic. Network traffic statistics (sum or average over a certain period) (Traffic) are collected from the Kafka cluster and aggregated separately, and can be calculated by "sum (bytes in) * len (replicas) + sum (bytes out)".

[0081] Through this structure, we will look at a first adjustment embodiment of leader balancing that takes network usage into account on a leader balancing system (1) of a usage-based topic partition in a Kafka cluster according to an embodiment of the present invention.

[0082] That is, in the first step, since the information on the individual components (Broker1, Broker2, Broker3) of each Kafka broker server (100) corresponding to the list of all brokers (Brokers) of the Kafka cluster in the above-described FIG. 3 is structured, the leader balance control module (200) traverses the broker list of the Kafka broker server (100) to find the Kafka broker server (100) having the maximum / minimum value of the sum {sum (Assumed Traffic)} of the network traffic estimates of each Kafka broker server (100) and checks the difference.

[0083] Here, the parameter of maximum value (MAX) is Bid MAX (maximum Kafka broker server id), SATR MAX (expressed as {maximum value of sum(Assumed Traffic) of Kafka broker server}, and the minimum value (MIN) parameter is Bid MIN (Minimum Kafka broker server id), SATR MIN It can be expressed as {the minimum value of sum(Assumed Traffic) of the Kafka broker server}. And the difference between the maximum and minimum values ​​of sum(Assumed Traffic) is SATR DIFF = SATR MAX - SATR MIN can be expressed as

[0084] In the second step, the leader balance control module (200) is SATR DIFF If the set threshold (e.g., 10%) is lower than the set threshold, the adjustment operation can be stopped. Another example of the present invention is "(SATR DIFF / SATR MAX )*100% < 10% → can be defined as "stop".

[0085] In the next third step, the leader balance control module (200) Bid MAXA leader partition (Leader) that meets the following conditions can be selected from the leader partition list (Leaders) of the Kafka broker server (100).

[0086] The conditions for selecting the leader partition are ① No-Move Topic (TP) BL ) is not, ② the number of replicas is 2 or more, ③ SATR DIFF The leader balance control module (200) may select the partition corresponding to the partition with the largest value among the assumed traffic values ​​less than / 2.

[0087] In the next fourth step, the leader balance control module (200) selects the selected partition (P SEL ) to Bid MIN Go to the Kafka broker server (100).

[0088] More specifically, the leader balance control module (200) selects a selected partition (P SEL ) to Bid MAX Remove from the leader partition list (Leaders) of the Kafka broker server (100), and Bid MAX Recalculate SATR{value for sum(Assumed Traffic)} for the corresponding Kafka broker server (100) and select the partition (P SEL ) to Bid MIN Add to the Leaders partition list of Bid MIN The SATR{value for sum(Assumed Traffic)} for the corresponding Kafka broker server (100) can be recalculated.

[0089] In the next fifth step, the leader balance control module (200) stops working if there are no moved partitions.

[0090] In the final sixth step, the leader balance control module (200) can repeat the entire process of steps 1 to 5 as many times as the set maximum number of repetitions.

[0091] Hereafter, we will look at a leader balancing method for usage-based topic partitions in a Kafka cluster on a leader balancing system (1) for usage-based topic partitions in a Kafka cluster according to a second regulation embodiment.

[0092] In the second moderation embodiment, a leveling step can be performed to prevent unnecessary additional leader imbalance adjustments by adjusting the Kafka configuration within the leader.imbalance.per.broker. percentage.

[0093] In the first step, the leader balance control module (200) traverses the broker list of the Kafka broker server (100) of FIG. 3 to find the Kafka broker server (100) that is the broker having the maximum / minimum network traffic estimate (Assumed Traffic) for each leader partition (Leader) constituting the leader partition list (Leaders) of the third category structure among the category structures, and checks the difference value.

[0094] Here, the parameter of maximum value (MAX) is Bid MAX (maximum Kafka broker server id), LDR MAX ({Maximum network traffic estimate (Assumed Traffic) for the Leader partition (Leader)} is expressed, and the minimum (MIN) parameter is Bid MIN (Minimum Kafka broker server id), LDR MIN (It can be expressed as {Minimum network traffic estimate (Assumed Traffic) for the leader partition (Leader)}. And the difference between the maximum and minimum values ​​of the network traffic estimate (Assumed Traffic) is LDR DIFF = LDR MAX - LDR MIN can be expressed as

[0095] In the second step, the leader balance control module (200) is LDR DIFFIf it is below the set threshold (e.g., 10%), the adjustment operation is stopped. Another expression example of the present invention is "(LDR DIFF / LDR MAX )*100% < 10% → can be defined as "stop".

[0096] In the third step, the leader balance control module (200) Bid MAX A leader partition that meets the following conditions is selected from the leader partition list (Leaders) of the corresponding Kafka broker server (100).

[0097] Under those conditions, the leader balance control module (200) can select a partition that is ① not a no-go topic (TPBL), ② has 2 or more replicas, and ③ has the smallest network traffic (Assumed Traffic) value.

[0098] In the fourth step, the leader balance control module (200) selects the selected partition (P SEL ) to Bid MIN Go to the Kafka broker server (100).

[0099] More specifically, the leader balance control module (200) selects a selected partition (P SEL ) to Bid MAX Remove from the leader partition list (Leaders) of the Kafka broker server (100) and select the selected partition (P SEL ) to Bid MIN can be added to the Leaders list of partitions.

[0100] Next, in the fifth step, the leader balance control module (200) stops working if there are no moved partitions.

[0101] In the final sixth step, the leader balance control module (200) can repeat the entire process of steps 1 to 5 as many times as the set maximum number of repetitions.

[0102] Meanwhile, FIGS. 5 and 6 are diagrams showing a method for adjusting leader balance of usage-based topic partitions in a Kafka cluster according to an embodiment of the present invention.

[0103] Referring to FIG. 5, the first embodiment of the leader balancing method of the usage-based topic partition in the Kafka cluster according to the embodiment of the present invention includes a process (S11) of traversing the broker list to find the broker with the maximum / minimum value of sum (Assumed Traffic) and checking the difference between them, SATR DIFF If the Bid is below the set threshold (e.g. 10%), the adjustment operation is stopped (S12). MAX The process of selecting a partition that meets the conditions from the Leaders of the broker (S13), the selected partition (P SEL ) to Bid MIN By comprising a process of moving to a broker (S14), a process of stopping work if there is no moved partition (S15), and a process of repeating the entire process for a set maximum number of repetitions (S16), the first to sixth steps of the first adjustment embodiment of leader balance adjustment considering network usage utilizing a leader balance adjustment system (1) of a usage-based topic partition in a Kafka cluster according to the embodiment of the present invention described above can be performed.

[0104] Also, referring to FIG. 6, the second embodiment of the leader balancing method of a usage-based topic partition in a Kafka cluster according to an embodiment of the present invention includes a process (S21) of traversing the broker list to find a broker with the maximum / minimum value of leaders and checking the difference between them, LDR DIFF If the Bid is below the set threshold (e.g. 10%), the adjustment operation is stopped (S22). MAX The process of selecting a partition that meets the conditions from the Leaders of the broker (S23), the selected partition (P SEL ) to Bid MINBy comprising a process of moving to a broker (S24), a process of stopping work if there is no moved partition (S25), and a process of repeating the entire process for a set maximum number of repetitions (S26), the first to sixth steps of the second adjustment embodiment of leader balance adjustment considering network usage utilizing a leader balance adjustment system (1) of a usage-based topic partition in a Kafka cluster according to the embodiment of the present invention described above can be performed.

[0105] The present invention can also be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system.

[0106] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and also those implemented in the form of carrier waves (e.g., transmission over the Internet).

[0107] Additionally, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be readily inferred by programmers in the technical field to which the present invention pertains.

[0108] As described above, the present specification and drawings have disclosed preferred embodiments of the present invention. Although specific terms have been used, they are used in a general sense only to easily explain the technical contents of the present invention and to assist in understanding the invention, and are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical concept of the present invention are possible in addition to the embodiments disclosed herein.

Claims

1. Multiple Kafka broker servers (100); and By traversing the broker list of the Kafka broker server (100) above, the Kafka broker server (100) with the maximum / minimum value of the sum of network traffic estimates {sum (Assumed Traffic)} of each Kafka broker server (100) is found and the difference is checked, and SATR DIFF If it is below the set threshold, the adjustment operation is stopped and the Bid MAX A leader partition (Leader) that meets the selection criteria is selected from the leader partition list (Leaders) of the Kafka broker server (100), and the selected partition (P) is selected. SEL ) to Bid MIN A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by including a leader balancing module (200) moving to a Kafka broker server (100); 2. In claim 1, the leader balance control module (200) A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by performing abort operations when no partitions have been moved.

3. In claim 2, the leader balance control module (200) A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by repeating the process a set maximum number of iterations.

4. Multiple Kafka broker servers (100); and By traversing the broker list of the Kafka broker server (100) above, the Kafka broker server (100) with the maximum / minimum network traffic estimate (Assumed Traffic) for each leader partition (Leader) constituting the leader partition list (Leaders) is found and the difference value is checked, and LDR DIFF If it is below the set threshold, the adjustment operation is stopped and the Bid MAX Select a leader partition that meets the selection conditions from the leader partition list (Leaders) of the Kafka broker server (100) corresponding to the selected partition (P SEL ) to Bid MIN A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by including a leader balancing module (200) moving to a Kafka broker server (100); 5. In claim 4, the leader balance control module (200) A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by performing abort operations when no partitions have been moved.

6. In claim 5, the leader balance control module (200) A leader balancing system for usage-based topic partitions in a Kafka cluster, characterized by performing a set maximum number of iterations.

7. The first step is to search for a Kafka broker server (100) having a maximum / minimum value of the sum of network traffic estimates {sum (Assumed Traffic)} of each Kafka broker server (100) by traversing the broker list of the Kafka broker server (100) and check the difference; SATR DIFF The second step is to stop the adjustment operation if it falls below the set threshold; Bid MAX A third step of selecting a leader partition (Leader) that meets the selection conditions from the leader partition list (Leaders) of the Kafka broker server (100); and Selected Partition (P SEL ) to Bid MIN A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized by comprising a fourth step of moving to a Kafka broker server (100); 8. In claim 7, after the fourth step, A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized in that it further includes a fifth step of performing a job abort if there is no moved partition; 9. In claim 8, after the fifth step, A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized in that the entire process of steps 1 to 5 is repeated a set maximum number of repetitions.

10. The first step of searching for a Kafka broker server (100) that has a maximum / minimum network traffic estimate (Assumed Traffic) for each leader partition (Leader) that constitutes the leader partition list (Leaders) by traversing the broker list of the Kafka broker server (100) and checking the difference value; LDR DIFF A second step is to perform a stop operation if the value falls below the set threshold; Bid MAX A third step of selecting a leader partition that satisfies the selection conditions from the leader partition list (Leaders) of the Kafka broker server (100); and Selected Partition (P SEL ) to Bid MIN A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized by comprising a fourth step of moving to a Kafka broker server (100); 11. In claim 10, after the fourth step, A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized in that it further includes a fifth step of performing a job abort if there is no moved partition; 12. In claim 11, after the fifth step, A method for leader balancing of usage-based topic partitions in a Kafka cluster, characterized in that the entire process of steps 1 to 5 is repeated a set maximum number of repetitions.

Images