System and method for rearranging topic partition in kafka cluster

Abstract

The present invention relates to a system and a method for rearranging a topic partition in a Kafka cluster and, more specifically, to a system and a method for rearranging a topic partition in a Kafka cluster for rearranging a partition according to a predetermined criterion such as a storage space usage capacity of a Kafka broker server.

Description

System and method for relocating topic partitions in a Kafka cluster

[0001] The present invention relates to a topic partition relocation system and method in a Kafka cluster, and more particularly, to a topic partition relocation system and method in a Kafka cluster for relocating partitions according to set criteria such as storage space usage capacity of a Kafka broker server.

[0002] Apache Kafka is a highly scalable, high-throughput distributed messaging system that guarantees reliable and continuous message delivery. Kafka applies the producer-consumer problem and utilizes three system components—Producer, Consumer, and Broker—to manage large volumes of data. It partitions topics and stores them sequentially. This stored data is then sequentially delivered to consumers for efficient processing.

[0003] Message data coming into a Kafka cluster is stored in a logical structure called a topic within Kafka.

[0004] A topic can have one or more replicas, and each replica is stored on a different Kafka broker server, allowing for resilience against the failure of a specific server.

[0005] Additionally, a topic is actually composed of one or more partitions that are stored on physical storage devices.

[0006] Therefore, when you configure a replica for a topic, physically identical partitions are stored in the storage devices of one or more Kafka broker servers.

[0007] A Kafka cluster can have multiple topics, and each topic can have multiple partitions, so a single Kafka cluster can have tens of thousands of partitions.

[0008] The problems with this are examined below.

[0009] In a Kafka cluster, when creating a topic / partition, the partitions are configured to be evenly distributed across each Kafka broker server.

[0010] The number of partitions is the criterion for evenly distributing partitions in a Kafka cluster, and the reason is to evenly distribute storage usage and network usage.

[0011] Since the amount of messages written to each topic in a Kafka cluster is different, the storage usage and network usage are also different.

[0012] Additionally, each topic can have a different retention period, which can result in significant differences in storage usage.

[0013] Therefore, when using a Kafka cluster, the storage usage of each Kafka broker server may vary and the deviation may be very large.

[0014] Accordingly, we provide a console tool called kafka-reassign-partitions.sh that can rearrange topic partitions to address storage usage skew.

[0015] However, this tool does not provide the ability to rearrange partitions based on the total storage capacity used.

[0016] You can manually redistribute partitions based on storage usage, but this is a nearly impossible task, as the number of topic partitions in a Kafka cluster can run into the tens of thousands.

[0017] Accordingly, in the relevant technical field, there is a need for technological development to rearrange partitions according to established criteria such as storage space usage capacity of Kafka broker servers.

[0018] [Prior Art Literature]

[0019] (Patent Document 1) Republic of Korea Patent Application No. 10-2017-0027318 (2017.03.02) "System and Method for Large Scale Image Processing in Real-Time Environments Using Apache Kafka"

[0020] (Patent Document 2) Republic of Korea Patent Application No. 10-2018-0136731 (November 8, 2018) "System and method for transmission of data in real time using multiple KAFKA"

[0021] The present invention is intended to solve the above problems, and to provide a topic partition relocation system and method in a Kafka cluster for relocating partitions according to established criteria such as storage space usage capacity of a Kafka broker server.

[0022] In addition, the present invention provides a topic partition relocation system and method in a Kafka cluster to provide even storage usage across multiple Kafka broker servers in the Kafka cluster.

[0023] In addition, the present invention provides a system and method for relocating topic partitions in a Kafka cluster so that they can be relocated with even network usage across multiple Kafka broker servers in the Kafka cluster.

[0024] However, the purposes of the present invention are not limited to the purposes mentioned above, and other purposes not mentioned will be clearly understood by those skilled in the art from the description below.

[0025] In order to achieve the above purpose, a topic partition relocation system in a Kafka cluster according to an embodiment of the present invention may provide a topic partition relocation system in a Kafka cluster, characterized in that it includes a plurality of Kafka broker servers (100); and a topic partition relocation module (200) that relocates partitions based on "storage usage capacity (usage) criteria" and "storage usage space ratio (usage rate) criteria" as a partition relocation method of each Kafka broker server (100).

[0026] At this time, the topic partition relocation module (200) can provide a topic partition relocation system in a Kafka cluster characterized in that it traverses the broker list (Brokers) to check the maximum / minimum storage usage values ​​for each individual component, the Kafka broker server (100), finds the Kafka broker server (100) that is the broker with the maximum / minimum storage usage values, checks the difference, and relocates the partition.

[0027] In addition, the topic partition rearrangement module (200) is, in the case of usage criteria,

[0028] MAX: Bid MAX , Sum of Partition Log Storage Usage (SPL) MAX )

[0029] MIN: Bid MIN , SPL MIN

[0030] SPL DIFF = SPL MAX - SPL MIN

[0031] A topic partition relocation system can be provided in a Kafka cluster characterized by checking the difference through an operation corresponding to .

[0032] In addition, the topic partition rearrangement module (200) is based on the usage rate (=used space (SPL) / total space (TLD)).

[0033] MAX: Bid MAX, Partition Log Storage Utilization (SPLR) MAX )

[0034] MIN: Bid MIN , SPLR MIN

[0035] SPLR DIFF = SPLR MAX - SPLR MIN

[0036] A topic partition relocation system can be provided in a Kafka cluster characterized by checking the difference through an operation corresponding to .

[0037] Additionally, the topic partition relocation module (200) can provide a topic partition relocation system in a Kafka cluster, characterized in that it additionally performs a step of equalizing the usage rate between each log directory within each Kafka broker server (100).

[0038] In addition, the topic partition relocation module (200) can provide a topic partition relocation system in a Kafka cluster characterized in that it traverses the log directories (Logdirs) for each Kafka broker server (100) to find the log directories (Logdirs) having the maximum / minimum storage usage rate (= sum of partition log sizes / log directory size) and relocates the partitions by checking the difference.

[0039] In order to achieve the above purpose, a topic partition relocation method in a Kafka cluster according to an embodiment of the present invention can provide a topic partition relocation method in a Kafka cluster, characterized in that it includes a first step in which a topic partition relocation module (200) collects storage usage capacity (usage) and storage usage space ratio (usage rate) for a plurality of Kafka broker servers (100); and a second step in which the topic partition relocation module (200) relocates partitions based on "storage usage capacity (usage) criteria" and "storage usage space ratio (usage rate) criteria" as a partition relocation method for each Kafka broker server (100).

[0040] At this time, in the second step, a topic partition relocation module (200) can provide a topic partition relocation method in a Kafka cluster, characterized in that it traverses a broker list (Brokers) to check the maximum / minimum storage usage values ​​for each individual component, a Kafka broker server (100), finds a Kafka broker server (100) that is a broker with a maximum / minimum storage usage value, checks the difference, and relocates the partition.

[0041] In addition, the second step, if the topic partition rearrangement module (200) is based on usage,

[0042] MAX: Bid MAX , Sum of Partition Log Storage Usage (SPL) MAX )

[0043] MIN: Bid MIN , SPL MIN

[0044] SPL DIFF = SPL MAX - SPL MIN

[0045] A method for rearranging topic partitions in a Kafka cluster can be provided, characterized by checking the difference through an operation corresponding to .

[0046] In addition, the second step, when the topic partition rearrangement module (200) is based on the usage rate (=used space (SPL) / total space (TLD)),

[0047] MAX: Bid MAX , Partition Log Storage Utilization (SPLR) MAX )

[0048] MIN: Bid MIN , SPLR MIN

[0049] SPLR DIFF = SPLR MAX - SPLR MIN

[0050] A method for rearranging topic partitions in a Kafka cluster can be provided, characterized by checking the difference through an operation corresponding to .

[0051] In addition, a method for rearranging topic partitions in a Kafka cluster can be provided, characterized in that after the first or second step, the topic partition rearrangement module (200) performs a step of leveling the usage rate between each log directory within each Kafka broker server (100).

[0052] In addition, the topic partition relocation module (200) can provide a topic partition relocation method in a Kafka cluster, characterized in that it traverses the log directories (Logdirs) for each Kafka broker server (100) to find the log directories (Logdirs) having the maximum / minimum storage usage rate (= sum of partition log sizes / log directory size) and relocates the partitions by checking the difference.

[0053] A topic partition relocation system and method in a Kafka cluster according to an embodiment of the present invention provides the effect of relocating partitions according to set criteria such as storage space usage capacity of a Kafka broker server.

[0054] In addition, a topic partition relocation system and method in a Kafka cluster according to another embodiment of the present invention provides an effect of providing even storage usage to multiple Kafka broker servers in a Kafka cluster.

[0055] In addition, the topic partition relocation system and method in a Kafka cluster according to another embodiment of the present invention provides the effect of enabling relocation with even network usage across multiple Kafka broker servers in a Kafka cluster.

[0056] FIG. 1 is a diagram illustrating a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention.

[0057] FIG. 2 is a diagram for explaining the relationship between an existing Kafka broker server-topic-partition and partition for configuring a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention.

[0058] FIG. 3 is a diagram showing a data structure provided by a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention.

[0059] FIG. 4 is a diagram showing the structure of a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention.

[0060] FIG. 5 and FIG. 6 are diagrams illustrating a topic partition rearrangement method in a Kafka cluster according to an embodiment of the present invention.

[0061] Hereinafter, a detailed description of preferred embodiments of the present invention will be provided with reference to the attached drawings. In the following description of the present invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the present invention.

[0062] In this specification, when a component 'transmits' data or a signal to another component, it means that the component can transmit the data or signal directly to the other component, or can transmit the data or signal to the other component via at least one other component.

[0063] FIG. 1 is a diagram illustrating a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention. Referring to FIG. 1, the topic partition relocation system (1) in the Kafka cluster configures a Kafka cluster composed of a plurality of Kafka broker servers (100), and message data entering the Kafka cluster is stored in a logical structure called a topic within Kafka.

[0064] That is, a large amount of data can be stored in order by setting topics and configuring partitions based on the topics.

[0065] At this time, one topic can have one or more replicas, and each replica is stored in a different Kafka broker server (100), so that failure of a specific Kafka broker server (100) can be prepared.

[0066] Here, a topic is actually composed of one or more partitions that are stored on physical storage devices.

[0067] Therefore, when replicas are configured for a topic, physically identical partitions are stored in one or more Kafka broker servers (100).

[0068] One of the multiple replicas of this partition is treated as the original and is called the leader partition, while the rest are called follower partitions.

[0069] Messages entering the Kafka cluster are added to the end of the leader partition and are read by follower partitions and message consumers. Therefore, all write / read operations occur in the Kafka broker server (100) that has the leader partition.

[0070] A Kafka cluster can have multiple topics, and each topic can have multiple partitions, so a single Kafka cluster can have tens of thousands of partitions.

[0071] FIG. 2 is a diagram illustrating the relationship between a conventional Kafka broker server, topic, partition, and partition for configuring a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention. Referring to FIG. 2, when a topic is added while using a Kafka cluster, partitions can be distributed as in the example of FIG. 2.

[0072] In the embodiment of FIG. 2, the number of brokers in the Kafka cluster, i.e., the number of Kafka broker servers (100), is 3, the number of topics is 3, A, B, and C, topic A is configured with 3 partitions / 2 replicas, topic B is configured with 4 partitions / 1 replica, and topic C can be configured with 2 partitions / 3 replicas.

[0073] In order to resolve the imbalance in storage device usage according to multiple Kafka broker servers (100) in a Kafka cluster structure such as this, a topic partition relocation system and method in a Kafka cluster for relocating partitions according to set criteria such as storage space usage capacity (usage, usage rate) of the Kafka broker servers (100) rather than manual work will be examined in detail below.

[0074] Fig. 3 is a diagram showing a data structure provided by a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention. Fig. 4 is a diagram showing the structure of a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention.

[0075] First, referring to FIG. 4, the leader balancing system (1) of a usage-based topic partition in a Kafka cluster may have a structure in which a leader balancing module (200) is added to the existing structure of FIG. 1.

[0076] Next, referring to FIG. 3, a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention can relocate partitions based on the following various criteria as a new partition relocation method to solve the problem.

[0077] The first criterion corresponds to the “storage usage capacity (usage) criterion”, and the topic partition relocation system (1) in the Kafka cluster according to the embodiment of the present invention can be used when the storage capacity of each Kafka broker server (100) is the same or similar.

[0078] The second criterion corresponds to the "storage usage space ratio (usage rate) criterion", and the topic partition relocation system (1) in the Kafka cluster according to the embodiment of the present invention can be used when the storage capacity of each Kafka broker server (100) is different from each other.

[0079] The effects that can be achieved by rearranging partitions in this way are as follows:

[0080] The first effect corresponds to the even storage usage across multiple Kafka broker servers (100) in a Kafka cluster.

[0081] Additionally, there is a second effect corresponding to the even network usage across multiple Kafka broker servers (100) in a Kafka cluster.

[0082] Next, referring to FIG. 3, a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention can provide a data structure corresponding to a topic partition relocation method to solve an existing problem.

[0083] As a first category structure, it structures information on individual components (Broker1, Broker2, Broker3) of each Kafka broker server (100) corresponding to the list of all brokers (Brokers) of the Kafka cluster.

[0084] Next, as a second category structure, information on individual components (Broker1, Broker2, Broker3) corresponding to each Kafka broker server (100), which is an individual component of the first category structure, may include ① broker ID (Broker.id), ② log directory (Logdirs), ③ total log directory storage space sum {sum(log_dir_size)}, and ④ broker partition log size sum {sum(sum(partition_log_size))}.

[0085] More specifically, the broker ID (Broker.id) corresponds to a unique ID of the Kafka broker server (100), the log directory (Logdirs) corresponds to a directory list for storing message logs of the Kafka broker server (100), the sum of the entire storage space of the log directory {sum(log_dir_size)} corresponds to the sum of the entire storage space of the log directory (Logdirs), and the sum of the broker partition log sizes {sum(sum(partition_log_size))} has information about the sum of the message log sizes of all partitions owned by each Kafka broker server (100) {sum(sum(partition_log_size))}.

[0086] Next, in the third category structure, there is information about individual directories (Logdir) included in each log directory (Logdirs) belonging to each Kafka broker server (100) of the second category structure, and may include ① path (path), ② partition list (Partitions), ③ total log directory storage space (log_dir_size), and ④ sum of log directory partition log sizes (sum(partition_log_size)).

[0087] Here, the sum of the log directory partition log sizes (sum(partition_log_size)) corresponds to the sum of the message log sizes of all partitions stored in each log directory.

[0088] Next, in the fourth category structure, there is information about each partition (Logdir) included in each partition list (Partitions) belonging to each individual directory (Logdir) of the third category structure, and may include ① topic name (topic), ② partition number (partition number), and ③ message log size of the partition (log_size).

[0089] Through these first to fourth category structures, an embodiment of the first relocation step of a topic partition relocation method performed on a topic partition relocation system (1) in a Kafka cluster according to an embodiment of the present invention will be examined.

[0090] As the first step of the first relocation phase, the topic partition relocation module (200) traverses the broker list (Brokers) to check the maximum / minimum storage usage for each individual component, the Kafka broker server (100), to find the Kafka broker server (100) that is the broker with the maximum / minimum storage usage and checks the difference.

[0091] In the case of the (1) usage criterion corresponding to the first embodiment of the first stage, the topic partition rearrangement module (200) can check the difference through the following operation.

[0092] MAX: Bid MAX , Sum of Partition Log Storage Usage (SPL) MAX )

[0093] MIN: Bid MIN , SPL MIN

[0094] SPL DIFF = SPL MAX - SPL MIN

[0095] Meanwhile, in the case of the (2) usage rate (=used space (SPL) / total space (TLD)) corresponding to the second embodiment of the first stage, the topic partition rearrangement module (200) can check the difference through the following operation.

[0096] MAX: Bid MAX , Partition Log Storage Utilization (SPLR) MAX )

[0097] MIN: Bid MIN , SPLR MIN

[0098] SPLR DIFF = SPLR MAX - SPLR MIN

[0099] Next, in the second step of the first relocation step, if the storage usage difference is less than a preset threshold (e.g., 10%), the topic partition relocation module (200) stops the relocation operation.

[0100] In the case of the (1) usage criterion corresponding to the first embodiment of the second stage, the topic partition rearrangement module (200) can check the difference through the following operation.

[0101] (SPL DIFF / SPL MAX )×100% < preset threshold (e.g. 10%) → stop

[0102] In the case of the (2) usage rate criterion corresponding to the second embodiment of the second stage, the topic partition rearrangement module (200) can check the difference through the following operation.

[0103] SPLR DIFF < Preset threshold (e.g. 10%) → Stop

[0104] Next, in the third step of the first relocation step, the Bid of the first step MAX Bid corresponding to the Kafka broker server (100) MAX The topic partition rearrangement module (200) selects a partition that satisfies the following conditions from the partition list (partitions) corresponding to the third category structure corresponding to the log directory (Logdirs) corresponding to the second category structure of the broker's drawing 3.

[0105] That is, the topic partition rearrangement module (200) is configured to ① pre-designated prohibited topics (TP) in the partition. BL ) not, ② Bid MAX There is a Kafka broker server (100) corresponding to the broker, but Bid MIN A partition that does not exist in the Kafka broker server (100) corresponding to the broker, ③ a partition that is below the preset usage standard or usage rate standard can be selected.

[0106] More specifically, Bid MAX Although it is in the broker, Bid MIN Looking at the partitions that are not in the broker, the topic partition relocation module (200) may move the target broker Bid to the partition that is to be moved, as the partition may have more than one replica depending on the settings. MIN Because the corresponding partition may already exist.

[0107] Meanwhile, when selecting a partition corresponding to a preset usage criterion or usage rate criterion, if it is a usage criterion, the topic partition rearrangement module (200) SPL DIFF You can select the partition with the largest message log size among the partitions with a usage size smaller than / 2.

[0108] And, when selecting a partition corresponding to a preset usage criterion or usage rate criterion, if it is a usage rate criterion, the topic partition rearrangement module (200) (SPLR DIFF *TLD MIN ) / 2 You can select the partition with the largest message log size among the partitions with a size smaller than .

[0109] Next, in the fourth step of the first relocation phase, the selected partition (P SEL ) Topic partition relocation module (200) is Bid MIN Move to broker.

[0110] That is, the topic partition rearrangement module (200) ① selects the partition (P SEL ) to Bid MAX Remove from Kafka broker server (100) corresponding to broker, ② Bid MAX Recalculate the Kafka broker server (100) usage / usage (SPL) corresponding to the broker, ③ Selected partition (P SEL ) to Bid MIN Add to Kafka broker server (100) corresponding to broker, ④ Bid MIN Recalculate the usage / capacity (SPL) of the Kafka broker server (100) corresponding to the broker.

[0111] Next, in the fifth step of the first relocation step, the topic partition relocation module (200) stops working if there are no moved partitions.

[0112] Finally, as the sixth step of the first relocation step, the topic partition relocation module (200) can repeat the entire process of steps 1 to 5 of the first relocation step a preset maximum number of repetitions.

[0113] Meanwhile, through the category structure of FIG. 3, the topic partition relocation system (1) in the Kafka cluster according to the embodiment of the present invention will be examined for an embodiment of the second relocation step of the topic partition relocation method.

[0114] Here, the second relocation step can be performed after the first relocation step.

[0115] The second relocation step corresponds to a step in which the topic partition relocation module (200) equalizes the usage rate between each log directory within each Kafka broker server (100).

[0116] As the first step of the second relocation phase, the topic partition relocation module (200) traverses the log directories (Logdirs) among the second category structures of FIG. 3 for each Kafka broker server (100) to find the log directories (Logdirs) with the maximum / minimum storage usage rate (= partition log size sum / log directory size) and checks the difference between them, but as in the following formula, the maximum and minimum log directory usage rate (SPLR) MAX , SPLR MIN ) SPLR corresponding to the car DIFF can be utilized.

[0117] MAX: Log directory size (LDS MAX ), log directory path (LP MAX ), log directory usage (SPLR MAX )

[0118] MIN: Log directory size (LDS MIN ), log directory path (LP MIN ,), log directory usage (SPLR MIN )

[0119] SPLR DIFF = SPLR MAX - SPLR MIN

[0120] In the second step of the second relocation phase, the topic partition relocation module (200) stops the relocation operation if the difference in storage usage is less than a preset threshold (e.g., less than 10%).

[0121] In the third step of the second relocation phase, the topic partition relocation module (200) sets the log directory path (LP MAX ) Select a partition that meets the following conditions in Partitions.

[0122] That is, the topic partition rearrangement module (200) ① prohibits movement of topics (TP BL ) not a partition, ② Maximum size of a movable partition (PS MAX ) = (SPLR DIFF × LDS MIN ) / 2 or less, ③ Maximum size of the movable partition (PS MAX) ) Select the largest partition among the partitions that do not exceed .

[0123] In the fourth step of the second relocation step, the topic partition relocation module (200) ① selects the partition (P SEL ) to the log directory path (LP MIN ) Go to ② Log directory path (LP MAX ) Selected partition (P) from Partitions SEL ) to remove ③ Log directory path (LP MAX ) Recalculate the usage rate / usage (SPL) ④ Log directory path (LP MIN ) in the selected partition (P) SEL ) Add ⑤ Log directory path (LP MIN ) continuously performs the process of recalculating the usage rate / usage (SPL).

[0124] In the fifth step of the second relocation step, the topic partition relocation module (200) performs a work stoppage if there is no moved partition.

[0125] In the sixth step of the second relocation step, the topic partition relocation module (200) repeatedly performs the entire process of steps 1 to 5 of the second relocation step a preset maximum number of repetitions.

[0126] Next, FIGS. 5 and 6 are diagrams illustrating a topic partition rearrangement method in a Kafka cluster according to an embodiment of the present invention.

[0127] First, referring to FIG. 5, the topic partition relocation method in the Kafka cluster according to the embodiment of the present invention corresponds to the first relocation step described above, which traverses the broker list (Brokers) to find the Kafka broker server (100) that is the broker with the maximum / minimum storage usage value and checks the difference (S11), and if the storage usage difference is less than a preset threshold, the topic partition relocation module (200) stops the relocation task (S12), and Bid MAX Bid corresponding to the Kafka broker server (100) MAX Select a partition (partition) that satisfies the conditions from the list of partitions corresponding to the broker's log directory (Logdirs) (S13), and select the selected partition (P SEL ) to Bid MIN The process of moving to a broker (S14), stopping the operation if there is no moved partition (S15), and repeating the entire process for the set maximum number of repetitions (S16) may be performed identically or similarly to steps 1 to 6 of the first relocation step described above.

[0128] Next, referring to FIG. 6, the topic partition relocation method in the Kafka cluster according to the embodiment of the present invention corresponds to the second relocation step described above, and searches the log directories (Logdirs) for each Kafka broker server (100) to find the log directories (Logdirs) having the maximum / minimum storage usage rate (= partition log size sum / log directory size) and checks the difference (S21), and if the storage usage rate difference is less than a preset threshold, the relocation task is stopped (S22), and LP MAX Select a partition that meets the conditions from the Partitions (S23), and select the selected partition (P SEL ) can be performed in the same or similar manner as steps 1 to 6 of the second relocation step described above, such as moving (S24), stopping the operation if there is no moved partition (S25), and repeating the entire process for the set maximum number of repetitions (S26).

[0129] The present invention can also be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system.

[0130] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and also those implemented in the form of carrier waves (e.g., transmission over the Internet).

[0131] Additionally, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be readily inferred by programmers in the technical field to which the present invention pertains.

[0132] As described above, the present specification and drawings have disclosed preferred embodiments of the present invention. Although specific terms have been used, they are used in a general sense only to easily explain the technical contents of the present invention and to assist in understanding the invention, and are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical concept of the present invention are possible in addition to the embodiments disclosed herein.

Claims

1. Multiple Kafka broker servers (100); and A topic partition relocation system in a Kafka cluster, characterized by including a topic partition relocation module (200) that relocates partitions based on “storage usage capacity (usage) criteria” and “storage usage space ratio (usage rate) criteria” as a partition relocation method of each Kafka broker server (100).

2. In claim 1, The above topic partition relocation module (200) is A topic partition relocation system in a Kafka cluster, characterized in that it traverses a list of brokers (Brokers) to check the maximum / minimum storage usage for each individual component, a Kafka broker server (100), finds a Kafka broker server (100) that is a broker with a maximum / minimum storage usage, and relocates the partition by checking the difference.

3. In claim 2, The above topic partition relocation module (200) is, in the case of usage criterion, MAX : Bid MAX , Sum of Partition Log Storage Usage (SPL) MAX ) MIN : Bid MIN , SPL MIN SPL DIFF = SPL MAX - SPL MIN A topic partition reorganization system in a Kafka cluster, characterized by checking the difference through a corresponding operation.

4. In claim 2, The above topic partition rearrangement module (200) is based on the usage rate (=used space (SPL) / total space (TLD)). MAX : Bid MAX , Partition Log Storage Utilization (SPLR) MAX ) MIN : Bid MIN , SPLR MIN SPLR DIFF = SPLR MAX - SPLR MIN A topic partition reorganization system in a Kafka cluster, characterized by checking the difference through a corresponding operation.

5. In claim 1, The above topic partition relocation module (200) is A topic partition relocation system in a Kafka cluster, characterized by additionally performing a step of equalizing utilization between each log directory within each Kafka broker server (100).

6. In claim 5, The above topic partition relocation module (200) is A topic partition relocation system in a Kafka cluster, characterized in that it traverses the log directories (Logdirs) for each Kafka broker server (100) to find the log directories (Logdirs) with the maximum / minimum storage usage ratio (= sum of partition log sizes / log directory size), and relocates the partitions by checking the difference.

7. A first step in which the topic partition relocation module (200) collects storage usage capacity (usage) and storage usage space ratio (usage rate) for multiple Kafka broker servers (100); and A topic partition rearrangement method in a Kafka cluster, characterized in that the topic partition rearrangement module (200) includes a second step of rearranging partitions based on “storage usage capacity (usage) criteria” and “storage usage space ratio (usage rate) criteria” as a partition rearrangement method of each Kafka broker server (100).

8. In claim 7, In the second step above, A topic partition rearrangement method in a Kafka cluster, characterized in that the above topic partition rearrangement module (200) traverses a broker list (Brokers) to check the maximum / minimum storage usage for each individual component, a Kafka broker server (100), finds a Kafka broker server (100) that is a broker with the maximum / minimum storage usage, checks the difference, and rearranges the partition.

9. In claim 8, The second step above is, If the above topic partition relocation module (200) is based on usage, MAX : Bid MAX , Sum of Partition Log Storage Usage (SPL) MAX ) MIN : Bid MIN , SPL MIN SPL DIFF = SPL MAX - SPL MIN A method for rearranging topic partitions in a Kafka cluster, characterized by checking the difference through a corresponding operation.

10. In claim 8, The second step above is, If the above topic partition relocation module (200) is based on the usage rate (=used space (SPL) / total space (TLD)), MAX : Bid MAX , Partition Log Storage Utilization (SPLR) MAX ) MIN : Bid MIN , SPLR MIN SPLR DIFF = SPLR MAX - SPLR MIN A method for rearranging topic partitions in a Kafka cluster, characterized by checking the difference through a corresponding operation.

11. In claim 7, After the above step 1 or step 2, The above topic partition relocation module (200) A method for relocating topic partitions in a Kafka cluster, characterized by performing a step of equalizing utilization between each log directory within each Kafka broker server (100).

12. In claim 11, The above topic partition relocation module (200) is A method for rearranging topic partitions in a Kafka cluster, characterized by traversing the log directories (Logdirs) for each Kafka broker server (100) to find the log directories (Logdirs) with the maximum / minimum storage usage ratio (= sum of partition log sizes / log directory size), and rearranging the partitions by checking the difference.

Images