A partition expansion method, device, storage medium and computer equipment of Kafka

Abstract

The present invention discloses a partition expansion method, device, storage medium and computer equipment of Kafka, which can determine the subject to be partition expanded in Kafka, obtain the copy distribution information of the current partition of the subject in the node of Kafka, obtain the current load information of the node of Kafka, determine the copy deployment plan of the partition to be expanded of the subject according to the obtained copy distribution information and current load information, and deploy the copy of the partition to be expanded in the node of Kafka according to the copy deployment plan to expand the partition to be expanded of the subject. The present invention can deploy the copy to the node with low current load value and reasonable copy distribution, avoid deploying the copy to the node with too high load or unreasonable copy distribution, effectively reduce the load distribution imbalance and data imbalance in each node, thereby reducing the imbalance of resource consumption between each node and improving the resource utilization of the node.

Description

Technical Field

[0001] The present invention relates to the technical field of Kafka, and in particular to a partition expansion method, device, storage medium and computer equipment of Kafka. Background Art

[0002] Kafka is a high-throughput distributed publish-subscribe messaging system that is often used in the field of big data. Kafka can be set up with multiple topics, each topic can be set up with multiple partitions, and a partition can be set up with a master copy and multiple slave copies. Among them, the copies of a topic can be distributed in each node of the Kafka cluster.

[0003] As the amount of data increases, the existing technology can expand the partitions of Kafka topics to meet consumers' data consumption needs and increase the number of concurrent data consumption. When expanding the partitions of the topic to be expanded, the number of copies of the partition to be expanded can be determined first, and then the copies of the expanded partitions can be deployed to the nodes of the Kafka cluster respectively.

[0004] However, when deploying each copy of the expanded partition to the nodes of the Kafka cluster respectively, the prior art usually randomly deploys each copy to one or more nodes of the Kafka cluster, which may cause an uneven distribution of all copies of the topic to be partitioned expanded in the relevant nodes, thereby causing an unbalanced load in the relevant nodes, resulting in an unbalanced resource consumption between the relevant nodes, and affecting the service performance of the Kafka cluster. Summary of the invention

[0005] In view of the above problems, the present invention provides a partition expansion method, device, storage medium and computer device of Kafka that overcome the above problems or at least partially solve the above problems. The technical solution is as follows:

[0006] A partition expansion method for Kafka, comprising:

[0007] Determine the topic in the Kafka to be partitioned and expanded;

[0008] Obtain the replica distribution information of the current partition of the topic in the nodes of the Kafka;

[0009] Obtain the current load information of the Kafka node;

[0010] Determine a replica deployment plan for the area to be expanded of the topic according to the obtained replica distribution information and the current load information;

[0011] According to the replica deployment scheme, a replica of the to-be-expanded region is deployed in the node of the Kafka to expand the to-be-expanded region of the topic.

[0012] Optionally, the replica distribution information includes primary replica distribution information, and the replica deployment plan includes a primary replica deployment plan.

[0013] Optionally, determining the replica deployment scheme of the to-be-expanded zone of the topic according to the obtained replica distribution information and the current load information includes:

[0014] Determine from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit; if so, determine, based on the master-replica distribution information, whether there is a node in the node whose load is not greater than the preset load limit where the master replica of the topic is not deployed; if so, determine the node where the master replica of the topic is not deployed as the target deployment node of the master replica of the partition to be expanded;

[0015] Deploying the replica of the to-be-expanded zone in the node of the Kafka according to the replica deployment scheme includes:

[0016] The master replica of the zone to be expanded is deployed in the target deployment node of the master replica.

[0017] Optionally, determining the replica deployment scheme of the to-be-expanded zone of the topic according to the obtained replica distribution information and the current load information further includes:

[0018] If there is no node on which the master copy of the topic is not deployed among the nodes whose load is not greater than the preset load limit, then, based on the master copy distribution information, determine whether there is a first node among the nodes whose load is not greater than the preset load limit and whose number of deployments of master copies of the topic is not higher than a preset threshold, and if so, determine the first node as the target deployment node for the master copy.

[0019] Optionally, the replica distribution information includes: slave replica distribution information, and the replica deployment plan includes: slave replica deployment plan.

[0020] Optionally, determining the replica deployment scheme of the to-be-expanded zone of the topic according to the obtained replica distribution information and the current load information includes:

[0021] Determine from the current load information at least one node among the nodes of the Kafka whose load is not greater than a preset load limit;

[0022] Determine the slave replica distribution information of the subject in each node whose load is not greater than a preset load limit;

[0023] Based on the deployment principle of minimizing the variance of the number of deployed slave replicas of the topic at each node whose load is not greater than the preset load limit, according to the slave replica distribution information of the topic in each node whose load is not greater than the preset load limit, determine the target deployment node of each slave replica of the topic from each node whose load is not greater than the preset load limit;

[0024] Deploying the replica of the to-be-expanded zone in the node of the Kafka according to the replica deployment scheme includes:

[0025] Deploy each slave replica of the topic to the corresponding target deployment node.

[0026] Optionally, the current load information includes: at least one of storage space occupancy, network bandwidth occupancy, central processing unit CPU usage and memory usage.

[0027] A partition expansion device for Kafka includes: a topic determination unit, a first acquisition unit, a second acquisition unit, a scheme determination unit and a replica deployment unit, wherein:

[0028] The topic determination unit is configured to execute: determining a topic to be partitioned and expanded in the Kafka;

[0029] The first obtaining unit is configured to execute: obtaining the replica distribution information of the current partition of the topic in the nodes of the Kafka;

[0030] The second obtaining unit is configured to execute: obtaining current load information of the Kafka node;

[0031] The scheme determination unit is configured to execute: determining a replica deployment scheme for the to-be-expanded zone of the topic according to the obtained replica distribution information and the current load information;

[0032] The replica deployment unit is configured to execute: deploying a replica of the to-be-expanded region in the node of the Kafka according to the replica deployment scheme, so as to expand the to-be-expanded region on the topic.

[0033] Optionally, the replica distribution information includes primary replica distribution information, and the replica deployment plan includes a primary replica deployment plan.

[0034] Optionally, the solution determination unit includes: a first determination unit, a second determination unit and a third determination unit, wherein:

[0035] The first determining unit is configured to execute: determining from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit, and if yes, triggering the second determining unit;

[0036] The second determining unit is configured to execute: determining, according to the primary-replica distribution information, whether there is a node on which the primary replica of the subject is not deployed among the nodes whose load is not greater than the preset load limit, and if so, triggering the third determining unit;

[0037] The third determining unit is configured to execute: determining the node where the primary replica of the subject is not deployed as the target deployment node of the primary replica of the zone to be expanded;

[0038] The replica deployment unit is configured to execute: deploying the primary replica of the to-be-expanded zone in the target deployment node of the primary replica.

[0039] Optionally, the solution determination unit further includes: a fourth determination unit and a fifth determination unit, wherein:

[0040] The first determining unit is configured to execute: determining from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit, and if not, triggering the fourth determining unit;

[0041] The fourth determination unit is configured to perform: determining, according to the primary-replica distribution information, whether there is a first node whose number of deployments of primary replicas of the topic is not higher than a preset threshold among the nodes whose loads are not greater than a preset load limit, and if so, triggering the fifth determination unit;

[0042] The fifth determining unit is configured to execute: determining the first node as a target deployment node for the primary replica.

[0043] Optionally, the replica distribution information includes: slave replica distribution information, and the replica deployment plan includes: slave replica deployment plan.

[0044] Optionally, the solution determination unit includes: a sixth determination unit, a seventh determination unit and an eighth determination unit, wherein:

[0045] The sixth determining unit is configured to execute: determining, from the current load information, at least one node among the nodes of the Kafka whose load is not greater than a preset load limit;

[0046] The seventh determining unit is configured to execute: determining the slave replica distribution information of the subject in each node whose load is not greater than a preset load limit;

[0047] The eighth determination unit is configured to execute: based on the deployment principle of minimizing the variance of the number of deployments of the slave replicas of the subject by the nodes whose loads are not greater than the preset load limit, according to the slave replica distribution information of the subject in the nodes whose loads are not greater than the preset load limit, determining the target deployment node of each slave replica of the subject from the nodes whose loads are not greater than the preset load limit;

[0048] The replica deployment unit is configured to execute: deploying each slave replica of the subject to a corresponding target deployment node.

[0049] Optionally, the current load information includes: at least one of storage space occupancy, network bandwidth occupancy, central processing unit CPU usage and memory usage.

[0050] A storage medium stores computer executable instructions, which, when loaded and executed by a processor, implement any of the above-mentioned Kafka partition expansion methods.

[0051] A computer device comprises a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are implemented:

[0052] Determine the topic in the Kafka to be partitioned and expanded;

[0053] Obtaining the replica distribution information of the topic in the nodes of the Kafka;

[0054] Obtain the current load information of the Kafka node;

[0055] Determine a replica deployment plan for the area to be expanded of the topic according to the obtained replica distribution information and the current load information;

[0056] According to the replica deployment scheme, a replica of the to-be-expanded region is deployed in the node of the Kafka to expand the to-be-expanded region of the topic.

[0057] The partition expansion method, device, storage medium and computer equipment of Kafka proposed in this embodiment can determine the topic to be partitioned in Kafka, obtain the copy distribution information of the current partition of the topic to be partitioned in the node of Kafka, obtain the current load information of the node of Kafka, determine the copy deployment plan of the partition to be expanded of the topic to be partitioned according to the obtained copy distribution information and current load information, and deploy the copy of the partition to be expanded in the node of Kafka according to the copy deployment plan to expand the partition to be expanded for the topic to be partitioned. The present invention can deploy the copy to the node with low current load value and reasonable copy distribution, avoid deploying the copy to the node with too high load or unreasonable copy distribution, effectively reduce the load distribution imbalance and data imbalance in each node, thereby reducing the imbalance of resource consumption between nodes, improving the resource utilization of nodes, thereby reducing the difficulty of Kafka cluster operation and maintenance, and improving the service performance of Kafka cluster. The present invention can determine the replica deployment plan before expanding the partition to be expanded, that is, according to the replica distribution information of the current partition of the topic to be partition expanded in the Kafka node and the current load information of the Kafka node. Before the problem of data imbalance occurs, the distribution plan of the replicas of the partition to be expanded in the Kafka node can be completed, which can effectively avoid the occurrence of data imbalance problems, and can expand the partition without affecting the service performance of the Kafka cluster and the service capacity of the topic of the expanded partition, thereby reducing the difficulty and risk of solving data balance.

[0058] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 A flowchart of a Kafka partition expansion method provided by an embodiment of the present invention is shown;

[0061] Figure 2 A flowchart of another Kafka partition expansion method provided by an embodiment of the present invention is shown;

[0062] Figure 3 A flowchart of another Kafka partition expansion method provided by an embodiment of the present invention is shown;

[0063] Figure 4 A flowchart of another Kafka partition expansion method provided by an embodiment of the present invention is shown;

[0064] Figure 5 A schematic structural diagram of a Kafka partition expansion device provided by an embodiment of the present invention is shown;

[0065] Figure 6 A schematic structural diagram of another Kafka partition expansion device provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0066] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0067] like Figure 1 As shown, this embodiment proposes a partition expansion method for Kafka, which may include the following steps:

[0068] S100. Determine the topic in Kafka that is to be partitioned and expanded.

[0069] Among them, the present invention can be applied to electronic devices, such as mobile phones, tablet computers, desktop computers, etc.

[0070] It should be noted that Kafka can be set with one or more topics. The present invention can set different topic identifiers for each topic in Kafka to distinguish and identify each topic according to the topic identifier.

[0071] The present invention does not limit the specific composition of the subject identifier. For example, the subject identifier can be an identifier composed of at least one or more of letters, numbers, and symbols.

[0072] Optionally, the present invention can allow a technician to input the subject identifier of the subject to be partitioned and determine the subject to be partitioned based on the subject identifier input by the technician; optionally, the present invention can also accept a partition expansion instruction sent by other electronic devices and carrying the subject identifier of the subject to be partitioned, and determine the subject to be partitioned based on the subject identifier in the partition expansion instruction.

[0073] In practical applications, the present invention can also obtain the total number of partitions of the subject to be partitioned expanded after the partition expansion is input by a technician or other electronic device, and determine the size between the current total number of partitions of the subject to be partitioned expanded and the above total number of partitions after the partition expansion. If the current total number of partitions of the subject to be partitioned expanded is less than the above total number of partitions after the partition expansion, the present invention can continue to execute the subsequent steps of partition expansion. Otherwise, the present invention can determine that the subject does not need to be partitioned expanded, and can prohibit the execution of subsequent steps of partition expansion, thereby reducing unnecessary consumption of partition expansion resources.

[0074] S200. Obtain the replica distribution information of the current partition of the topic to be partition expanded in the Kafka nodes.

[0075] The current partition may be a partition currently set in Kafka for a certain topic. Accordingly, the current partition of the topic to be partitioned may be a partition currently set in Kafka for the topic to be partitioned.

[0076] It should be noted that the present invention does not limit the number of current partitions of the topic to be partitioned, for example, the current number of partitions can be one, or two. The present invention can set different partition identifiers for each partition in Kafka to distinguish and identify each partition in Kafka according to the partition identifier. The present invention does not limit the specific composition of the partition identifier, for example, the partition identifier can be an identifier composed of at least one or more of letters, numbers, symbols, etc.

[0077] The nodes in Kafka may be nodes in a Kafka cluster. The present invention may set different node identifiers for each node in Kafka to distinguish and identify each node in Kafka according to the node identifier. The present invention does not limit the specific composition of the node identifier. For example, the node identifier may be an identifier composed of at least one or more of letters, numbers, and symbols.

[0078] The replica distribution information may be the distribution information of the replicas of the current partition of a certain topic in the nodes of Kafka. Correspondingly, the replica distribution information of the topic to be partitioned may be the distribution information of the replicas of the current partition of the topic to be partitioned in the nodes of Kafka.

[0079] Optionally, the replica distribution information may include the distribution information of the master replica of the current partition of a certain topic in the nodes of Kafka. For example, for the topic to be partitioned, its replica distribution information may include: the master replica of the first partition of the topic to be partitioned is distributed in the node whose node identifier is the first node identifier, and the master replica of the second partition of the topic to be partitioned is distributed in the node whose node identifier is the second node identifier. Optionally, the replica distribution information may include the distribution information of the slave replicas of the current partition of a certain topic in the nodes of Kafka. Correspondingly, for the topic to be partitioned, its replica distribution information may include: a slave replica of the first partition of the topic to be partitioned is distributed in the node whose node identifier is the third node identifier, two slave replicas of the first partition of the topic to be partitioned are distributed in the node whose node identifier is the fourth node identifier, and two slave replicas of the second partition of the topic to be partitioned are distributed in the node whose node identifier is the fifth node identifier.

[0080] Optionally, the present invention can set different identifiers for each replica in Kafka to distinguish and identify each replica in Kafka according to the replica identifier, for example, the master replica of the first partition of the first topic is identified as the first master replica, and the two slave replicas of the first partition of the first topic are identified as the first slave replica and the second slave replica. Among them, the present invention does not limit the specific composition of the replica identifier. For example, the replica identifier can be an identifier composed of at least one or more of letters, numbers, and symbols. At this time, the present invention can obtain the distribution information of the replicas of different replica identifiers in the topic to be partitioned and expanded in the nodes of Kafka. For example, for the topic to be partitioned and expanded, its replica distribution information may include: the first master replica of the first partition of the topic to be partitioned and expanded is distributed in the node whose node identifier is the first node identifier, the first slave replica of the first partition of the topic to be partitioned and expanded is distributed in the node whose node identifier is the second node identifier, and the second slave replica and the third slave replica of the first partition of the topic to be partitioned and expanded are distributed in the node whose node identifier is the third node identifier.

[0081] Specifically, the present invention can send a data acquisition instruction for acquiring replica distribution information of a topic to be partitioned and expanded to each node in the Kafka cluster, so as to obtain the replica distribution information of the topic to be partitioned and expanded returned by the relevant node.

[0082] In an optional embodiment, the copy distribution information of the current partition of each topic in Kafka can be stored in a certain data storage space. Specifically, the present invention can store the copy distribution information of the topic identifier and the current partition of the topic in the Kafka node in the data storage space. At this time, after determining the topic identifier of the topic to be partitioned, the present invention can search and obtain the copy distribution information of the topic to be partitioned in the Kafka node from the data storage space.

[0083] S300. Obtain current load information of Kafka nodes.

[0084] The load information may be the usage of computing resources, data transmission resources, storage resources, etc. by the node. For example, the load information may be the storage space occupancy rate, network bandwidth occupancy rate, or central processing unit (CPU) usage rate.

[0085] Optionally, in an optional embodiment, the current load information may include: at least one of storage space occupancy, network bandwidth occupancy, CPU usage and memory usage.

[0086] It should be noted that the present invention can obtain the current load information of a certain node of Kafka, determine the current remaining load capacity of the node according to the current load information of the node, and determine whether the node can continue to deploy replicas, so as to avoid deploying the replicas of the expanded partition to the node whose load has exceeded or is about to exceed the limit when the partition expansion is performed on the topic to be partitioned, thereby avoiding the possible situation where the load of the node is too high.

[0087] Optionally, the present invention can calculate the current load value of the node according to the current load information of the node, and evaluate the current remaining load capacity of the node according to the current load value of the node. It should be noted that when the current load value of a node is higher, the present invention can consider that the current remaining load capacity of the node is lower; when the current load value of a node is lower, the present invention can consider that the current remaining load capacity of the node is higher.

[0088] Among them, the present invention can assign corresponding weights to the usage rates of various types of resources in the current load information of the node, and then substitute the usage rates and corresponding weights of various types of resources into the selected load value calculation formula (such as the average value calculation formula, the weighted calculation formula, etc.) to calculate the current load value of the node. For example, when the current load information includes storage space occupancy, network bandwidth occupancy, CPU usage and memory usage, the present invention can assign 25% weights to the storage space occupancy, network bandwidth occupancy, CPU usage and content usage, select the following weighted calculation formula as the load value calculation formula, substitute the usage rates and corresponding weights of various types of resources into the following weighted calculation formula, and calculate the current load value of the node:

[0089] A=A1*w1+A2*w2+A3*w3+A4*w4

[0090] Among them, A is the load value, A1 is the storage space occupancy, A2 is the network bandwidth occupancy, A3 is the CPU usage, A4 is the memory usage, w1 is the weight corresponding to the storage space occupancy, w2 is the weight corresponding to the network bandwidth occupancy, w3 is the weight corresponding to the CPU usage, and w4 is the weight corresponding to the memory usage.

[0091] It should be noted that the weights of the utilization rates of various types of resources and the calculation formulas of the load values ​​can be determined by technical personnel according to actual working conditions, and the present invention does not limit this.

[0092] Optionally, the present invention can obtain the current load information of each node of Kafka, and determine whether there is a node with a lower current load value among the nodes of Kafka, so that when deploying replicas of the partition to be expanded, it can be planned to deploy the replicas in the nodes with lower current load values ​​in priority, thereby improving the utilization rate of node resources.

[0093] Optionally, when the replicas of the partition to be expanded are specified to be deployed in a certain part of the nodes, the present invention can only obtain the current load information of the part of the nodes, and prohibit obtaining the current load information of other parts of the nodes, so as to avoid unnecessary resource consumption and improve resource utilization.

[0094] S400. Determine a replica deployment plan for the partition to be expanded of the subject to be partitioned based on the obtained replica distribution information and current load information.

[0095] It should be noted that, before determining the replica deployment scheme for the partition to be expanded, the present invention may first determine the number of replicas to be set for the partition to be expanded. For example, the number of replicas to be set for the partition to be expanded may be five, namely one master replica and four slave replicas.

[0096] Among them, the replica deployment plan can be a deployment plan that plans to deploy replicas of the partition to be expanded to the relevant nodes of Kafka. The replica deployment plan can store relevant information of the replica of the partition to be expanded (such as the replica type, the number of replicas, etc.) and the node identifier of the target deployment node of the replica. It should be noted that after the replica deployment plan for the partition to be expanded is determined, the present invention can determine the target deployment node of each replica in the partition to be expanded from the replica deployment plan. For example, the present invention can determine from the replica deployment plan that the target deployment node of the master replica of the partition to be expanded is the node identified by the first node, determine that the target deployment node of the first slave replica of the partition to be expanded is identified as the node identified by the second node, and determine that the target deployment node of the second slave replica and the third slave replica of the partition to be expanded is identified as the node identified by the third node.

[0097] Among them, in the process of determining the replica deployment plan for the partition to be expanded, the present invention can combine the distribution information of the current partition of the topic to be partitioned expanded in the Kafka nodes, and the current load information of the Kafka nodes, to determine the priority of each Kafka node in deploying the replica of the partition to be expanded, and generate a deployment plan for deploying the replicas of the partition to be expanded to the relevant nodes of Kafka according to the priority. For example, for a node that has not deployed a replica and has the lowest load value, the present invention can determine that the node is the node with the highest priority for deploying replicas of the partition to be expanded, and thus the node can be determined as a target deployment node for replicas of the partition to be expanded; for another example, for a node that has not deployed a primary replica of a subject to be partitioned and has the lowest load value, the present invention can determine that the node is a node with a higher priority for deploying the primary replica of the partition to be expanded, and thus the node can be determined as a target deployment node for the primary replica of the partition to be expanded; for another example, for a node that has fewer replicas of the partition to be expanded and has a load value that does not exceed the limit, the present invention can determine that the node is a target deployment node with a higher priority for deploying slave replicas of the partition to be expanded, and thus the node can be determined as a target deployment node for slave replicas of the partition to be expanded; for a node that has not deployed a replica but has a load value that exceeds the limit, the present invention can determine that the node is a target deployment node with a lower priority for deploying replicas of the partition to be expanded, and thus the node can be prohibited from being determined as a target deployment node for replicas of the partition to be expanded.

[0098] Optionally, the present invention can generate a replica deployment list corresponding to the relevant information of the replicas of the to-be-expanded partitions and the node identification of the target deployment node of the replicas according to the deployment plan formulated according to the priority level. It should be noted that the content of the relevant information of the replicas of the to-be-expanded partitions and the node identification of the target deployment node of the replicas in the replica deployment list corresponding to the stored information can be the replica deployment plan.

[0099] Among them, the plan of the present invention to deploy the replicas of the partition to be expanded to the nodes with lower current load values ​​can avoid deploying the replicas of the partition to be expanded to the nodes with higher current load values, avoid the situation where the nodes are overloaded, and also enable Kafka to have all the replicas of each topic more evenly distributed in the nodes of Kafka after the partition is expanded.

[0100] In practical applications, when a node is deployed with multiple master copies of the same topic, it may cause the node to consume a large amount of certain types of resources, such as network bandwidth. Therefore, in an optional embodiment, in the process of determining the replica deployment plan, the present invention can combine the distribution information of the current partition of the topic to be partitioned expanded in the nodes of Kafka, and the current load information of the nodes of Kafka, to determine the node where the master copy of the topic to be partitioned expanded is not deployed and the load value does not exceed the limit value, and determine the node as a node with a higher priority for planning to deploy the master copy of the partition to be expanded, thereby determining the node as the target deployment node for the master copy of the partition to be expanded.

[0101] It should be noted that, in the process of determining the replica deployment scheme, the present invention can first determine the target deployment node of a replica of the partition to be expanded, then simulate the situation of deploying the replica to the target deployment node, obtain the replica distribution information of the current partition of the topic to be partitioned expanded after the simulation in the Kafka node and the current load information of the Kafka node, and then determine the target deployment node of another replica of the partition to be expanded again. And so on, until the target deployment nodes of all replicas of the partition to be expanded are determined, so as to determine the replica deployment scheme of the partition to be expanded.

[0102] S500. According to the replica deployment plan, a replica of the partition to be expanded is deployed in the Kafka node to expand the partition to be expanded for the topic to be partitioned.

[0103] Among them, after determining the replica deployment plan of the partition to be expanded, the present invention can determine the target deployment nodes of each replica of the partition to be expanded from the replica deployment plan of the partition to be expanded, and deploy each replica of the partition to be expanded to the corresponding target deployment node to complete the partition expansion of the subject to be partitioned.

[0104] It should be noted that the deployment method of the present invention, in which the target deployment nodes of each replica of the partition to be expanded are determined and then each replica is deployed, can uniformly deploy each replica of the partition to be expanded to the corresponding target deployment node, which can effectively reduce the deployment resources consumed in the replica deployment process. However, in the process of determining the replica deployment plan, it is necessary to simulate the distribution of the replicas and the load conditions of the nodes, which increases the resource consumption required for the simulation. Determining the replica deployment plan in real time during the simulation can effectively improve the accuracy of the obtained replica distribution information and load information, thereby further avoiding the situation where the node is overloaded, so that all replicas of the topic to be partitioned and expanded can be more evenly distributed in the Kafka nodes.

[0105] Optionally, the present invention may also determine the relevant information of the replica of the partition to be expanded (such as the type and number of replicas) and the target deployment node as a replica deployment plan and execute step S500 each time determining the target deployment node of a replica of the partition to be expanded, until all replicas of the partition to be expanded are deployed to the corresponding target deployment node.

[0106] It should be noted that the deployment method of the present invention, in which a replica is deployed each time the target deployment node of a replica of the zone to be expanded is determined, requires executing the deployment process of the replica multiple times, which increases the resource consumption during the replica deployment process. However, there is no need to simulate the replica distribution and load information in the process of determining the replica deployment plan, thereby avoiding the resource consumption caused by the simulation.

[0107] In practical applications, when executing step S100, if the number of partitions to be expanded of the subject to be partitioned and expanded input by the technician or other electronic device is multiple, the present invention can expand the partitions to be expanded one by one, that is, steps S100, S200, S300, S400 and S500 can be executed for one of the partitions to be expanded, and then steps S100, S200, S300, S400 and S500 can be executed for the expansion of another partition to be expanded, until the expansion of each partition to be expanded is completed.

[0108] It should also be noted that when the prior art randomly deploys the replicas of the partition to be expanded to the nodes of Kafka to expand the partition to be expanded, it may cause data imbalance in the relevant nodes where the replicas of the topic of the expanded partition are deployed. The prior art can manually redistribute the replicas and migrate the data stored in the replicas according to the distribution of the replicas of the topic after the expansion of the partition and the resource consumption of the relevant nodes to solve the problem of data imbalance. However, the prior art can only manually redistribute the replicas and migrate the data after the data imbalance problem occurs, which belongs to post-data balancing. Since the replicas may have stored a large amount of data at this time, the manual redistribution of the replicas and the migration of the data will consume a large amount of network bandwidth and CPU of the relevant nodes, affecting the ability of the Kafka cluster to provide services, and increasing the difficulty and risk of manual replica redistribution and data migration. Among them, the manual method of replica allocation and data migration also requires a lot of manpower and time. However, before expanding the partition to be expanded, the present invention can determine the replica deployment plan according to the replica distribution information of the current partition of the topic to be partition expanded in the Kafka node and the current load information of the Kafka node, and complete the distribution plan of the replicas of the partition to be expanded in the Kafka node before the data imbalance problem occurs, which can effectively avoid the occurrence of data imbalance problem, and expand the partition without affecting the service performance of the Kafka cluster and the service capacity of the topic of the expanded partition, thereby reducing the difficulty and risk of solving data balance.

[0109] The partition expansion method of Kafka proposed in this embodiment can determine the topic to be partitioned in Kafka, obtain the copy distribution information of the current partition of the topic to be partitioned in the node of Kafka, obtain the current load information of the node of Kafka, determine the copy deployment plan of the partition to be expanded of the topic to be partitioned according to the obtained copy distribution information and current load information, and deploy the copy of the partition to be expanded in the node of Kafka according to the copy deployment plan, so as to expand the partition to be expanded for the topic to be partitioned. The partition expansion method can deploy the copy to the node with low current load value and reasonable copy distribution, avoid deploying the copy to the node with too high load or unreasonable copy distribution, effectively reduce the load distribution imbalance and data imbalance in each node, thereby reducing the imbalance of resource consumption between nodes, improving the resource utilization of nodes, thereby reducing the difficulty of Kafka cluster operation and maintenance, and improving the service performance of Kafka cluster. This partition expansion method can determine the replica deployment plan before expanding the partition to be expanded, that is, according to the replica distribution information of the current partition of the topic to be expanded in the Kafka node and the current load information of the Kafka node. Before the data imbalance problem occurs, the distribution plan of the replicas of the partition to be expanded in the Kafka node can be completed, which can effectively avoid the occurrence of data imbalance problems, and can expand the partition without affecting the service performance of the Kafka cluster and the service capacity of the topic of the expanded partition, thereby reducing the difficulty and risk of solving data balance.

[0110] based on Figure 1 The steps shown in this embodiment propose another partition expansion method of Kafka, such as Figure 2 In this method, the replica distribution information may include primary replica distribution information, and the replica deployment scheme may include primary replica deployment scheme. Figure 2 middle, Figure 1 The step S400 shown may include the following steps:

[0111] S401. Determine from the current load information whether there is a node in the Kafka node whose load is not greater than the preset load limit. If yes, execute step S402.

[0112] It should be noted that, for the master copy of the partition to be expanded, the present invention can plan to preferentially deploy it to the node whose current load value is not greater than the preset load limit and where the master copy of the topic to be partitioned expanded is not deployed. This can avoid deploying multiple master copies of the topic to be partitioned expanded on a certain node, reduce the production and consumption rate of the topic to be partitioned expanded, reduce the resource consumption of the node, and avoid deploying the master copy to the node with a higher current load value, thereby avoiding the load value of the node with a higher current load value exceeding the limit, and can improve the resource utilization of the node with a lower current load value.

[0113] The preset load limit value may be determined by a technician according to actual working conditions, and the present invention does not limit this.

[0114] Specifically, the present invention can determine the node identifier of each node whose current load value is not greater than the preset load limit from the current load information of each node of Kafka, so as to determine each node whose current load value is not greater than the preset load limit.

[0115] It should be noted that if there is no node in the Kafka nodes whose load is not greater than the preset load limit, that is, the current load of each Kafka node is greater than the preset load limit, then the present invention can prohibit the partition expansion of the topic to be partitioned, so as to avoid causing a greater operating burden on the Kafka nodes.

[0116] S402. Determine, based on the master-replica distribution information, whether there are any nodes whose loads are not greater than a preset load limit and which do not deploy the master replica of the subject to be partitioned and expanded. If so, execute step S403.

[0117] Specifically, the present invention can determine the nodes where the master copy of the topic to be partitioned expanded is not deployed from the nodes whose current load value is not greater than the preset load limit based on the master copy distribution information of the current partition of the topic to be partitioned expanded in each node.

[0118] It should be noted that if there is no node on which the primary copy of the topic to be partitioned expanded is not deployed among the nodes whose load is not greater than the preset load limit, that is, all the nodes whose load is not greater than the preset load limit are deployed with the primary copy of the topic to be partitioned expanded, then the present invention can prohibit partition expansion of the topic to be partitioned expanded, so as to avoid causing a greater operating burden on the Kafka nodes.

[0119] S403: Determine the node where the primary replica of the subject to be partitioned expanded is not deployed as the target deployment node of the primary replica of the partition to be expanded.

[0120] Specifically, the present invention can determine a node whose current load value is not greater than a preset load limit and on which no primary replica of a subject to be partitioned is deployed as a target deployment node for the primary replica of the partition to be expanded.

[0121] It should be noted that if the current load value is not greater than the preset load limit and there are multiple nodes where the primary copy of the subject to be partitioned and expanded is not deployed, the present invention can randomly select a node from them to be determined as the target deployment node of the primary copy of the partition to be expanded. The present invention can also determine the node with the smallest current load value as the target deployment node of the primary copy of the partition to be expanded, so as to maximize the utilization of node resources.

[0122] After determining the target deployment node of the master copy of the partition to be expanded, the present invention can deploy the master copy of the partition to be expanded. Figure 2 As shown, Figure 1 The step S500 shown may specifically be:

[0123] S501. Deploy a master copy of the partition to be expanded in the target deployment node of the master copy to expand the partition to be expanded for the subject to be partitioned.

[0124] Specifically, the present invention can deploy the master copy of the partition to be expanded to the corresponding target deployment node, so as to expand the partition to be expanded for the subject to be partitioned.

[0125] The Kafka partition expansion method proposed in this embodiment can determine the node whose current load value is not greater than the preset load limit and on which the primary copy of the topic to be partition expanded is not deployed as the target deployment node of the primary copy of the partition to be expanded. This can avoid deploying multiple primary copies of the topic to be partition expanded on a certain node, thereby reducing the resource consumption of the node. It can also avoid deploying the primary copy to the node with a higher current load value, thereby avoiding the load value of the node with a higher current load value from exceeding the limit, and can improve the resource utilization of the node with a lower current load value.

[0126] based on Figure 2 The steps shown in this embodiment propose another partition expansion method of Kafka, such as Figure 3 As shown. Figure 3 middle, Figure 1 The step S400 shown may also include the following steps:

[0127] S404. If there is no node on which the master copy of the subject to be partitioned expanded is not deployed among the nodes whose load is not greater than the preset load limit, determine, based on the master copy distribution information, whether there is a first node among the nodes whose load is not greater than the preset load limit and whose number of deployed master copies of the subject to be partitioned expanded is not higher than a preset threshold, and if so, execute step S405.

[0128] It should be noted that, when the present invention determines that there is no node on which the primary copy of the subject to be partitioned expanded is not deployed among the nodes whose load is not greater than the preset load limit, that is, when the nodes whose load is not greater than the preset load limit are all deployed with the primary copy of the subject to be partitioned expanded, the node with a smaller number of deployed primary copies of the subject to be partitioned expanded can be determined from the nodes whose load is not greater than the preset load limit, and determined as the target deployment node of the primary copy of the partition to be expanded.

[0129] Specifically, the present invention is implemented Figure 2 In step S402, i.e., executing the judgment based on the master-copy distribution information to determine whether there are any nodes on which the master copy of the subject to be partitioned expansion is not deployed among the nodes whose load is not greater than the preset load limit, if the judgment result is no, the step of executing the master-copy distribution information to determine whether there are any first nodes on which the number of deployed master copies of the subject to be partitioned expansion is not higher than the preset threshold among the nodes whose load is not greater than the preset load limit.

[0130] The preset threshold value may be determined by a technician according to the operation performance and actual working conditions of the node, and the present invention does not limit this.

[0131] S405: Determine the first node as the target deployment node of the primary replica.

[0132] Specifically, the present invention can determine a node whose load is not greater than a preset load limit and whose number of deployed primary replicas of the subject to be partitioned expanded is not greater than a preset threshold as a target deployment node for the primary replica of the partition to be expanded.

[0133] It should be noted that if there are multiple nodes whose load is not greater than the preset load limit and whose number of deployed primary copies of the topic to be partitioned expanded is not higher than the preset threshold, the present invention can randomly select a node from them to be determined as the target deployment node of the primary copy of the partition to be expanded, or can determine the node with the lowest current load value or the least number of deployed primary copies of the topic to be partitioned expanded as the target deployment node of the primary copy of the partition to be expanded.

[0134] The Kafka partition expansion method proposed in this embodiment can, when all nodes with a load not greater than a preset load limit are deployed with the primary copy of the topic to be partitioned expanded, determine from among the nodes with a load not greater than the preset load limit a node with a deployment number of primary copies of the topic to be partitioned expanded not greater than a preset threshold, and determine it as the target deployment node of the primary copy of the partition to be expanded, so as to avoid a certain node deploying more primary copies of the topic to be partitioned expanded, thereby avoiding imposing a greater load and operating pressure on the node.

[0135] based on Figure 1 The steps shown in this embodiment propose another partition expansion method of Kafka, such as Figure 4 In this method, the replica distribution information may include: from the replica distribution information, and the replica deployment plan includes: from the replica deployment plan. Figure 4 middle, Figure 1 The step S400 shown may include the following steps:

[0136] S410. Determine, from the current load information, at least one node among the Kafka nodes whose load is not greater than a preset load limit.

[0137] It should be noted that the present invention can deploy the replicas of the subject to be partitioned and expanded to nodes whose load is not greater than a preset load limit, so as to avoid deploying the replicas to nodes with higher current load values, thereby avoiding load value exceeding the limit of nodes with higher current load values ​​and improving resource utilization of nodes with lower current load values.

[0138] Specifically, the present invention can determine the node identifier of each node whose current load value is not greater than the preset load limit from the current load information of each node of Kafka, so as to determine each node whose current load value is not greater than the preset load limit.

[0139] S420. Determine the slave replica distribution information of the subject to be partitioned and expanded in each node whose load is not greater than a preset load limit.

[0140] Specifically, the present invention can determine the secondary replica distribution information of the subject to be partitioned and expanded in each node whose load is not greater than a preset load limit.

[0141] S430. Based on the deployment principle of minimizing the variance of the number of deployed replicas of the subject to be partitioned expanded in each node whose load is not greater than the preset load limit, according to the distribution information of the replicas of the subject to be partitioned expanded in each node whose load is not greater than the preset load limit, determine the target deployment node of each replica of the subject to be partitioned expanded from the nodes whose load is not greater than the preset load limit.

[0142] It should be noted that in order to ensure that the slave replicas of the subject to be partitioned expanded can be evenly distributed to the greatest extent possible among the nodes whose current load value is not greater than the preset load limit, thereby improving the resource utilization of the nodes, the present invention can formulate a slave replica deployment plan based on the principle that the variance of the number of slave replicas deployed after the slave replicas of the partition to be expanded are deployed on each node whose current load value is not greater than the preset load limit is minimized.

[0143] Specifically, after determining each node whose load is not greater than the preset load limit, the present invention can determine the current deployed number of slave replicas of the topic to be partitioned expanded for each node whose current load value is not greater than the preset load limit based on the obtained slave replica distribution information.

[0144] Optionally, the present invention can determine the target deployment node of the slave replicas of the to-be-expanded zone in each node whose load is not greater than a preset load limit one by one. For example, when the to-be-expanded zone needs to be equipped with a first slave replica, a second slave replica and a third slave replica, a total of three slave replicas, the present invention can first determine the target deployment node of the first slave replica, then determine the target deployment node of the second slave replica, and then determine the target deployment node of the third slave replica.

[0145] Specifically, in the process of determining the target deployment node of any slave replica of the partition to be expanded, the present invention can determine the node with the smallest currently deployed number of slave replicas of the subject to be partitioned expanded among the nodes whose load is not greater than the preset load limit as the target deployment node of the slave replica. For example, when determining the target deployment node of the first slave replica of the partition to be expanded, if there are three nodes whose load is not greater than the preset load limit, namely the first node, the second node and the third node, and the first node has 5 currently deployed slave replicas of the subject to be partitioned expanded, while the second node has 4 and the third node has 3, then the present invention can determine the third node as the target deployment node of the first slave replica of the partition to be expanded.

[0146] It should be noted that, after determining the target deployment node of a certain slave replica of the partition to be expanded, the present invention can first simulate the situation of deploying the slave replica to the target deployment node, obtain the slave replica distribution information of the current partition of the topic to be partitioned expanded after the simulation in the Kafka node and the current load information of the Kafka node, and then determine the target deployment node of another slave replica of the partition to be expanded.

[0147] After determining the target deployment node of the slave replica of the partition to be expanded, the present invention can deploy the slave replica of the partition to be expanded. Figure 4 As shown, Figure 1 The step S500 shown may specifically be:

[0148] S510: deploy each slave replica of the subject to be partitioned and expanded to a corresponding target deployment node, so as to expand the subject to be partitioned and expanded.

[0149] Specifically, the present invention can deploy each slave replica of the partition to be expanded to the corresponding target deployment node, so as to expand the partition to be expanded for the subject to be partitioned.

[0150] The Kafka partition expansion method proposed in this embodiment can determine the target deployment nodes of each slave replica of the partition to be expanded based on the principle that the variance of the number of slave replicas deployed after the slave replicas of the partition to be expanded are deployed on each node whose current load value is not greater than the preset load limit is minimized, so that the slave replicas of the topic to be partitioned can be distributed as evenly as possible among the nodes whose current load value is not greater than the preset load limit, which can avoid the situation where the node load is too high due to uneven distribution of slave replicas and can also improve the resource utilization of the node.

[0151] Based on the above steps, this embodiment proposes another partition expansion method of Kafka, in which the replica distribution information can simultaneously include: master replica distribution information and slave replica distribution information, and the replica deployment plan can simultaneously include: master replica deployment plan and slave replica deployment plan.

[0152] Among them, the present invention can determine the master copy deployment plan and the slave copy deployment plan of the partition to be expanded of the topic to be partitioned according to the master copy distribution information and the slave copy distribution information of the topic to be partitioned expanded in the Kafka nodes, and the current load information of the Kafka nodes.

[0153] Optionally, the present invention can be directly based on Figure 2 The steps shown determine the master-replica deployment plan for the partition to be expanded, that is, determine the target deployment node of the master-replica according to the master-replica distribution information of the current partition of the topic to be partitioned and expanded in the Kafka node and the current load information of the Kafka node.

[0154] Optionally, in addition to considering the master copy distribution information of the current partition of the subject to be partitioned expanded in the Kafka nodes and the current load information of the Kafka nodes, the present invention can also consider the slave copy distribution information of the current partition of the subject to be partitioned expanded in the Kafka nodes when determining the target deployment node of the master copy of the partition to be expanded. For example, when determining the target deployment node of the master copy of the partition to be expanded, for two nodes with the same current load value and neither of which has deployed the master copy of the subject to be partitioned expanded, the present invention can obtain the slave copy distribution information of the subject to be partitioned expanded in the two nodes, compare the number of slave copies of the subject to be partitioned expanded deployed on the two nodes, and determine the node with a smaller number of slave copies to be partitioned expanded deployed in the two nodes as the target deployment node of the master copy of the partition to be expanded with a higher priority.

[0155] Optionally, the present invention can be directly based on Figure 4 The steps shown determine the slave replica deployment plan for the partition to be expanded, that is, directly determine the target deployment node of the slave replica for the partition to be expanded based on the slave replica distribution information of the current partition of the topic to be partitioned and the current load information of the Kafka node.

[0156] Optionally, in addition to considering the slave replica distribution information of the current partition of the topic to be partitioned expanded in the Kafka nodes and the current load information of the Kafka nodes, the present invention can also consider the master replica distribution information of the current partition of the topic to be partitioned expanded in the Kafka nodes when determining the target deployment node of the slave replica to be partitioned expanded. Figure 4 Step S420 in the above method may specifically include: determining the master copy distribution information and the slave copy distribution information of the subject to be partitioned and expanded in each node whose load is not greater than a preset load limit;

[0157] Step S430 may be specifically as follows: based on the deployment principle of minimizing the variance of the number of deployed replicas of the topic to be partitioned expanded at each node whose load is not greater than a preset load limit, according to the master replica distribution information and slave replica distribution information of the topic to be partitioned expanded in each node whose load is not greater than the preset load limit, determine the target deployment node of each slave replica of the topic to be partitioned expanded from each node whose load is not greater than the preset load limit.

[0158] It should be noted that, when determining the target deployment node of the slave replica, the present invention can formulate a slave replica deployment plan based on the principle that the variance of the number of replicas deployed after the replicas of the partition to be expanded (including the master replica and the slave replica) are deployed on each node whose current load value is not greater than the preset load limit is the minimum. For example, in the process of determining the target deployment node of a slave replica of the partition to be expanded, the present invention can determine the node with the smallest number of currently deployed replicas of the subject to be partitioned expanded as the target deployment node of the slave replica among the nodes whose load is not greater than the preset load limit.

[0159] The Kafka partition expansion scheme proposed in this embodiment can determine the master copy deployment scheme and the slave copy deployment scheme of the partition to be expanded of the topic to be partitioned according to the master copy distribution information and the slave copy distribution information of the topic to be partitioned expanded in the Kafka nodes, and the current load information of the Kafka nodes, which can further improve the utilization rate of node resources and further enable the slave copies of the topic to be partitioned expanded to be evenly distributed to the greatest extent possible among the nodes whose current load value is not greater than the preset load limit, thereby avoiding the situation where the node load is too high due to uneven copy distribution.

[0160] based on Figure 1 The method shown in this embodiment proposes a partition expansion device for Kafka, such as Figure 5 As shown, the apparatus may include: a theme determination unit 100, a first obtaining unit 200, a second obtaining unit 300, a scheme determination unit 400 and a replica deployment unit 500, wherein:

[0161] The topic determination unit 100 is configured to execute: determining a topic in Kafka to be partitioned and expanded.

[0162] The first obtaining unit 200 is configured to execute: obtaining the replica distribution information of the current partition of the topic to be partition expanded in the nodes of Kafka.

[0163] The second obtaining unit 300 is configured to execute: obtaining current load information of the Kafka node.

[0164] The load information may be the usage of computing resources, data transmission resources, storage resources, and other resources by the node. For example, the load information may be the storage space occupancy rate, the network bandwidth occupancy rate, or the CPU usage rate.

[0165] Optionally, in an optional embodiment, the current load information may include: at least one of storage space occupancy, network bandwidth occupancy, CPU usage and memory usage.

[0166] The solution determination unit 400 is configured to execute: determining a replica deployment solution for a partition to be expanded of a topic to be partition expanded according to the obtained replica distribution information and current load information.

[0167] The replica deployment unit 500 is configured to execute: deploying replicas of the partitions to be expanded in the nodes of Kafka according to the replica deployment plan, so as to expand the partitions to be expanded for the topics to be partitioned.

[0168] The Kafka partition expansion device proposed in this embodiment can determine the replica deployment plan before expanding the partition to be expanded, that is, according to the replica distribution information of the current partition of the topic to be expanded in the Kafka node and the current load information of the Kafka node. Before the data imbalance problem occurs, the distribution plan of the replicas of the partition to be expanded in the Kafka node can be completed, which can effectively avoid the occurrence of data imbalance problems, and can expand the partition without affecting the service performance of the Kafka cluster and the service capacity of the topic of the expanded partition, thereby reducing the difficulty and risk of solving data balance.

[0169] based on Figure 5 The device shown in the figure, this embodiment proposes another partition expansion device of Kafka, such as Figure 6 In the device, the replica distribution information may include primary replica distribution information, and the replica deployment scheme may include primary replica deployment scheme. The scheme determination unit 400 may include: a first determination unit 401, a second determination unit 402, and a third determination unit 403, wherein:

[0170] The first determining unit 401 is configured to execute: determining from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit, and if so, triggering the second determining unit 402.

[0171] The second determination unit 402 is configured to execute: based on the master-replica distribution information, determine whether there are nodes with loads not greater than a preset load limit that do not deploy the master replica of the subject to be partitioned expanded, and if so, trigger the third determination unit 403.

[0172] The third determining unit 403 is configured to execute: determining a node where the primary replica of the subject to be partitioned expanded is not deployed as a target deployment node for the primary replica of the partition to be expanded.

[0173] The Kafka partition expansion device proposed in this embodiment can determine the node whose current load value is not greater than the preset load limit and on which the primary copy of the topic to be partitioned expanded is not deployed as the target deployment node of the primary copy of the partition to be expanded. This can avoid deploying multiple primary copies of the topic to be partitioned expanded on a certain node, thereby reducing the resource consumption of the node. It can also avoid deploying the primary copy to the node with a higher current load value, thereby avoiding exceeding the load value of the node with a higher current load value, and can improve the resource utilization of the node with a lower current load value.

[0174] based on Figure 6 The device shown in the embodiment proposes another partition expansion device of Kafka. In the device, the solution determination unit 400 also includes: a fourth determination unit and a fifth determination unit, wherein:

[0175] The first determining unit 401 is configured to execute: determining from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit; if not, triggering the fourth determining unit.

[0176] The fourth determination unit is configured to execute: based on the master-replica distribution information, determine whether there is a first node among the nodes whose load is not greater than the preset load limit and whose number of deployed master replicas of the topic to be partitioned expanded is not higher than the preset threshold; if so, trigger the fifth determination unit.

[0177] The fifth determining unit is configured to execute: determining the first node as a target deployment node for the primary replica of the zone to be expanded.

[0178] Specifically, the present invention can determine a node whose load is not greater than a preset load limit and whose number of deployed primary replicas of the subject to be partitioned expanded is not greater than a preset threshold as a target deployment node for the primary replica of the partition to be expanded.

[0179] The Kafka partition expansion device proposed in this embodiment can, when all nodes with a load not greater than a preset load limit are deployed with the primary copy of the topic to be partitioned expanded, determine the node whose number of deployed primary copies of the topic to be partitioned expanded is not greater than a preset threshold from the nodes with a load not greater than the preset load limit, and determine it as the target deployment node of the primary copy of the partition to be expanded, so as to avoid a node deploying more primary copies of the topic to be partitioned expanded, thereby avoiding imposing a greater load and operating pressure on the node.

[0180] based on Figure 5The device shown in the embodiment proposes another partition expansion device of Kafka. In the device, the replica distribution information may include: from the replica distribution information, and the replica deployment plan includes: from the replica deployment plan. At this time, the plan determination unit 400 may include: a sixth determination unit, a seventh determination unit and an eighth determination unit, wherein:

[0181] The sixth determining unit is configured to execute: determining, from the current load information, at least one node among the nodes of Kafka whose load is not greater than a preset load limit.

[0182] The seventh determining unit is configured to execute: determining the slave replica distribution information of the subject to be partitioned and expanded in each node whose load is not greater than a preset load limit.

[0183] The eighth determination unit is configured to execute: based on the deployment principle of minimizing the variance of the number of deployed slave replicas of the topic to be partitioned expanded at each node with a load not greater than a preset load limit, according to the slave replica distribution information of the topic to be partitioned expanded in each node with a load not greater than the preset load limit, determining the target deployment node of each slave replica of the topic to be partitioned expanded from the nodes with a load not greater than the preset load limit.

[0184] The Kafka partition expansion device proposed in this embodiment can determine the target deployment nodes of each slave replica of the partition to be expanded based on the principle that the variance of the number of slave replicas deployed after the slave replicas of the partition to be expanded are deployed on each node whose current load value is not greater than the preset load limit is minimized, so that the slave replicas of the topic to be partitioned can be distributed as evenly as possible among the nodes whose current load value is not greater than the preset load limit, which can avoid the situation where the node load is too high due to uneven distribution of slave replicas and can also improve the resource utilization of the node.

[0185] The Kafka partition expansion device provided in an embodiment of the present invention may include a processor and a memory. The above-mentioned topic determination unit 100, the first acquisition unit 200, the second acquisition unit 300, the scheme determination unit 400 and the replica deployment unit 500, etc. can all be stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0186] The processor includes a kernel, which calls the corresponding program unit from the memory. The kernel can be set to one or more, and the partition expansion of the subject to be partitioned is performed by adjusting the kernel parameters.

[0187] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0188] An embodiment of the present invention provides a storage medium on which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, any of the above-mentioned Kafka partition expansion methods is implemented.

[0189] An embodiment of the present invention provides a processor, which is used to run a program, wherein the partition expansion method of Kafka is executed when the program is running.

[0190] A computer device provided in an embodiment of the present invention may include a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:

[0191] Determine the topic in Kafka to be partitioned and expanded;

[0192] Get the replica distribution information of the current partition of the topic to be partitioned in the Kafka nodes;

[0193] Get the current load information of Kafka nodes;

[0194] Determine the replica deployment plan for the partition to be expanded for the topic to be partitioned based on the obtained replica distribution information and current load information;

[0195] According to the replica deployment plan, deploy replicas of the partitions to be expanded in the Kafka nodes to expand the partitions of the topics to be expanded.

[0196] The computer device in this embodiment can be a server, PC, PAD, mobile phone, etc.

[0197] A computer program product provided by an embodiment of the present invention, when executed on a data processing device, is suitable for executing a program for initializing the following method steps:

[0198] Determine the topic in Kafka to be partitioned and expanded;

[0199] Get the replica distribution information of the current partition of the topic to be partitioned in the Kafka nodes;

[0200] Get the current load information of Kafka nodes;

[0201] Determine the replica deployment plan for the partition to be expanded for the topic to be partitioned based on the obtained replica distribution information and current load information;

[0202] According to the replica deployment plan, deploy replicas of the partitions to be expanded in the Kafka nodes to expand the partitions of the topics to be expanded.

[0203] It should be understood by those skilled in the art that the embodiments of the present invention may provide methods, devices, equipment (systems) or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0204] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, devices (systems) or computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0207] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0208] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0209] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0210] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A partition expansion method for Kafka, characterized in that: The method is applied to an electronic device, comprising: Determine the topic in the Kafka to be partitioned and expanded; Obtain the replica distribution information of the current partition of the topic in the nodes of the Kafka; Obtain the current load information of the Kafka node; Determine a replica deployment plan for the area to be expanded of the topic according to the obtained replica distribution information and the current load information; The process of determining the replica deployment scheme of the to-be-expanded full zone of the subject includes: determining a target deployment node for a replica of the to-be-expanded full zone; simulating the deployment of the one replica to the target deployment node; reacquiring the replica distribution information of the current partition of the subject in the nodes of the Kafka and the current load information of the nodes of the Kafka after the simulation; determining the target deployment node for the next replica of the to-be-expanded full zone according to the replica distribution information of the current partition of the subject in the nodes of the Kafka and the current load information of the nodes of the Kafka after the simulation; looping through the above steps until the target deployment nodes of all replicas of the to-be-expanded full zone are determined, so as to determine the replica deployment scheme of the to-be-expanded full zone of the subject; deploying the replica of the to-be-expanded full zone in the nodes of the Kafka according to the replica deployment scheme, so as to expand the to-be-expanded full zone of the subject; The replica distribution information includes primary replica distribution information, and the replica deployment scheme includes primary replica deployment scheme; The step of determining the replica deployment scheme of the partition to be expanded of the topic according to the obtained replica distribution information and the current load information further includes: determining from the current load information whether there are nodes in the nodes of the Kafka with a load not greater than a preset load limit; if so, determining according to the primary replica distribution information whether there are nodes in the nodes with a load not greater than the preset load limit on which the primary replica of the topic is not deployed; if so, determining the nodes on which the primary replica of the topic is not deployed as target deployment nodes for the primary replica of the partition to be expanded; Deploying the replica of the to-be-expanded partition in the node of the Kafka according to the replica deployment plan includes: deploying the primary replica of the to-be-expanded partition in the target deployment node of the primary replica.

2. The method according to claim 1, characterized in that The step of determining the replica deployment scheme of the to-be-expanded zone of the topic according to the obtained replica distribution information and the current load information further includes: If there is no node on which the master copy of the topic is not deployed among the nodes whose load is not greater than the preset load limit, then, based on the master copy distribution information, determine whether there is a first node among the nodes whose load is not greater than the preset load limit and whose number of deployments of master copies of the topic is not higher than a preset threshold, and if so, determine the first node as the target deployment node for the master copy.

3. The method according to claim 1, characterized in that The replica distribution information includes: slave replica distribution information, and the replica deployment plan includes: slave replica deployment plan.

4. The method according to claim 3, characterized in that Determining the replica deployment scheme of the to-be-expanded zone of the subject according to the obtained replica distribution information and the current load information includes: Determine from the current load information at least one node among the nodes of the Kafka whose load is not greater than a preset load limit; Determine the slave replica distribution information of the subject in each node whose load is not greater than a preset load limit; Based on the deployment principle of minimizing the variance of the number of deployed slave replicas of the topic at each node whose load is not greater than the preset load limit, according to the slave replica distribution information of the topic in each node whose load is not greater than the preset load limit, determine the target deployment node of each slave replica of the topic from each node whose load is not greater than the preset load limit; Deploying the replica of the to-be-expanded zone in the node of the Kafka according to the replica deployment scheme includes: Deploy each slave replica of the topic to the corresponding target deployment node.

5. The method according to claim 1, characterized in that The current load information includes: at least one of storage space occupancy, network bandwidth occupancy, central processing unit (CPU) usage and memory usage.

6. A partition expansion device for Kafka, characterized in that: The device is applied to an electronic device and comprises: a theme determination unit, a first obtaining unit, a second obtaining unit, a scheme determination unit and a replica deployment unit, wherein: The topic determination unit is configured to execute: determining a topic to be partitioned and expanded in the Kafka; The first obtaining unit is configured to execute: obtaining the replica distribution information of the current partition of the topic in the nodes of the Kafka; The second obtaining unit is configured to execute: obtaining current load information of the Kafka node; The scheme determination unit is configured to execute: determining the replica deployment scheme of the to-be-expanded partition of the subject according to the obtained replica distribution information and the current load information; the process of determining the replica deployment scheme of the to-be-expanded partition of the subject includes: determining the target deployment node of a replica of the to-be-expanded partition; simulating the deployment of the one replica to the target deployment node; reacquiring the replica distribution information of the current partition of the subject in the nodes of the Kafka and the current load information of the nodes of the Kafka after the simulation; determining the target deployment node of the next replica of the to-be-expanded partition according to the replica distribution information of the current partition of the subject in the nodes of the Kafka and the current load information of the nodes of the Kafka after the simulation; looping through the above steps until the target deployment nodes of all replicas of the to-be-expanded partition are determined, so as to determine the replica deployment scheme of the to-be-expanded partition of the subject; The replica deployment unit is configured to execute: deploying a replica of the to-be-expanded region in the node of the Kafka according to the replica deployment scheme, so as to expand the to-be-expanded region for the topic; The replica distribution information includes primary replica distribution information, and the replica deployment scheme includes primary replica deployment scheme; The scheme determination unit includes: a first determination unit, a second determination unit and a third determination unit, wherein: The first determining unit is configured to execute: determining from the current load information whether there is a node in the Kafka node whose load is not greater than a preset load limit, and if yes, triggering the second determining unit; The second determining unit is configured to execute: determining, according to the primary-replica distribution information, whether there is a node on which the primary replica of the subject is not deployed among the nodes whose load is not greater than the preset load limit, and if so, triggering the third determining unit; The third determining unit is configured to execute: determining the node where the primary replica of the subject is not deployed as the target deployment node of the primary replica of the zone to be expanded; The replica deployment unit is configured to execute: deploying the primary replica of the to-be-expanded zone in the target deployment node of the primary replica.

7. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the Kafka partition expansion method as described in any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a processor, a memory and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the partition expansion method of Kafka as described in any one of claims 1 to 5 is implemented. ：

Images