Method and device for electing primary copy of Kafka partition, equipment, medium and product

Abstract

The invention relates to a Kafka partition primary copy election method and device, equipment, a medium and a product. The method comprises the following steps: when determining that partition primary copy election needs to be carried out, judging whether a partition needing primary copy election has a subordinate partition group or not; if yes, selecting a main partition group from the partition groups in the plurality of server nodes according to a preset partition group selection strategy; the partition group information of the selected main partition group is written into Zookeeper, so that a Kafka controller is indicated to update the partition group information of the main partition group stored in the Kafka controller through the Zookeeper, and after updating, the Kafka controller sends a first change request to a server node where the selected main partition group is located; wherein the first change request is used for requesting a server node where the selected main partition group is located, so that each slave copy of the selected main partition group is changed into a main copy; in this way, the problems that ZooKeeper loads are too large and a network is blocked due to partition primary copy election are solved.

Description

Technical Field

[0001] The present application relates to the field of Kafka cluster technology, and in particular to a method and apparatus, device, medium and product for selecting a primary replica of a Kafka partition. Background Art

[0002] Kafka is a distributed message streaming platform that supports multiple partitions, multiple copies and is based on Zookeeper coordination. It is also an open source message engine system based on the publish-subscribe model.

[0003] Kafka partition master election is usually performed when the master in the partition fails. That is, when the leader of one or more partitions in the Kafka cluster fails, a new master needs to be elected from the followers of the partition.

[0004] In traditional technologies, when Kafka has more than 10,000 partitions, the election of partition master replicas will cause problems such as excessive Zookeeper load and network congestion. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device, equipment, medium and product for selecting the primary replica of a Kafka partition that can solve the problems of excessive ZooKeeper load and network congestion caused by the election of the primary replica of the partition.

[0006] In a first aspect, an embodiment of the present application provides a method for electing a primary replica of a Kafka partition, the method comprising:

[0007] When it is determined that a partition master replica election is required, determine whether the partition that requires a master replica election has a subordinate partition group;

[0008] If so, a primary partition group is selected from the partition groups in the multiple server nodes according to a preset partition group election strategy;

[0009] The partition group information of the elected primary partition group is written into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located;

[0010] The first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

[0011] In one of the embodiments, before selecting a primary partition group from the partition groups in the plurality of server nodes according to a preset partition group election strategy, the method further includes:

[0012] Determine whether all slave copies of the partition group in multiple server nodes are valid;

[0013] If all slave copies of the partition group in each server node are valid, a step of selecting a master partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy is performed.

[0014] In one embodiment, the method further comprises:

[0015] If at least one slave replica is invalid, the server node where the invalid slave replica is located is removed from the multiple server nodes, and after the removal, a master partition group is elected from the partition groups in the remaining server nodes according to a preset partition group election strategy;

[0016] If there is at least one invalid slave replica in the partition group in each server node, the election is terminated.

[0017] In one embodiment, the method further comprises:

[0018] When Zookeeper detects that at least one of the multiple server nodes is down, it is determined that partition master replica election is required.

[0019] In one embodiment, the method further comprises:

[0020] If the partition that needs to elect a master replica does not belong to any partition group, the master replica is elected from the slave replicas in multiple server nodes according to the preset partition election strategy;

[0021] The partition information of the elected primary replica is written into Zookeeper, so as to instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located;

[0022] The second change request is used to request the server node where the elected master copy is located, so as to change the elected master copy from a slave copy to a master copy.

[0023] In one of the embodiments, when a partition that needs to perform primary replica election has a partition group to which it belongs, in the multiple server nodes, the partition group in each server node has a partition group level listener registered in Zookeeper;

[0024] When the partition that needs to perform master replica election does not have a partition group to which it belongs, among the multiple server nodes, the slave replica in each server node registers a partition-level listener in Zookeeper.

[0025] In a second aspect, an embodiment of the present application provides a device for electing a primary replica of a Kafka partition, the device comprising:

[0026] A group election module is used to determine whether the partition that needs to perform the primary replica election has a subordinate partition group when it is determined that the partition primary replica election needs to be performed. If so, a primary partition group is selected from the partition groups in the multiple server nodes according to a preset partition group election strategy;

[0027] An update change module, used for writing the partition group information of the elected primary partition group into Zookeeper, so as to instruct the Kafka controller through Zookeeper to update the partition group information of the primary partition group stored in the Kafka controller, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located;

[0028] The first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

[0029] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0032] The above-mentioned Kafka partition master replica election method and device, equipment, medium and product, when it is determined that a partition master replica election is required, first determine whether the partition that needs to perform the master replica election has a subordinate partition group; if the partition that needs to perform the master replica election has a subordinate partition group, then according to the preset partition group election strategy, elect a master partition group from the partition groups in multiple server nodes; then, write the partition group information of the elected master partition group into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the master partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected master partition group is located, wherein the first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

[0033] In this way, since the embodiment of the present application first implements the election of the primary partition group through the partition group divided into multiple partitions, and then directly obtains the elected primary replica according to the primary partition group. During the election, it is only necessary to elect according to the partition group. Compared with the traditional technology of electing the primary replica according to a single partition, the ZooKeeper load is greatly reduced, the network congestion is improved, and the election efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A flowchart of a method for electing a primary replica of a Kafka partition according to an embodiment of the present invention is provided;

[0036] Figure 2 One of the schematic diagrams of a Kafka cluster of an embodiment;

[0037] Figure 3 This is a second schematic diagram of a Kafka cluster according to an embodiment;

[0038] Figure 4 The third schematic diagram of a Kafka cluster of an embodiment;

[0039] Figure 5 The method for electing a primary replica of a Kafka partition in one embodiment also includes one of the flow charts;

[0040] Figure 6The method for electing a primary replica of a Kafka partition in an embodiment also includes a second flow chart;

[0041] Figure 7 The present invention is a structural block diagram of a Kafka partition master replica election device according to an embodiment. DETAILED DESCRIPTION

[0042] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0044] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.

[0045] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0046] It can be understood that “at least one” means one or more, “plurality” means two or more, and “at least a portion of an element” means a part or all of an element.

[0047] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0048] In an exemplary embodiment, referring to Figure 1, provides a method for electing a primary replica of a Kafka partition, which may include the following steps S102~S106.

[0049] S102: When it is determined that a partition master replica election is required, it is determined whether the partition for which the master replica election is required has a subordinate partition group.

[0050] Among them, the Kafka cluster may include multiple topics (Topic), each topic may include multiple partitions (Partition), these multiple partitions may be distributed on different server nodes (Broker); each partition may include multiple replicas (Replica).

[0051] Exemplary, reference Figure 2 , one of the multiple topics in the Kafka cluster includes 9 partitions, namely Partition1~ Partition9, each of which includes 4 replicas, which are distributed in Broker1~ Broker4. Taking Partition1 as an example, it is explained in detail: Partition1 includes 4 replicas, which are distributed in Broker1~ Broker4; Partition1 itself is counted in the 4 replicas of Partition1, in other words, Partition1 itself is one of the 4 replicas of Partition1, Partition1L is the primary replica selected from the 4 replicas of Partition1, and Partition1F is the slave replica from the 4 replicas of Partition1.

[0052] Exemplary, reference Figure 3 , in Broker1~Broker4, Partition1, Partition4 and Partition7 all have their own partition groups, and the partition groups they belong to are all partition group 1, that is, PartitionGroup1; Partition8 and Partition9 all have their own partition groups, and the partition groups they belong to are all partition group 2, that is, PartitionGroup2; Partition2, Partition3, Partition5 and Partition6 do not have their own partition groups, that is, they are not added to the partition group.

[0053] Continue to refer Figure 3, we can see that for PartitionGroup1, PartitionGroup1 exists in Broker1~ Broker4; when PartitionGroup1 in Broker1 is the master partition group (PartitionGroup1L) and PartitionGroup1 in Broker2~ Broker4 are all slave partition groups (PartitionGroup1F), the partitions of PartitionGroup1L are all master replicas, which are Partition1L, Partition4L and Partition7L respectively, and the partitions of PartitionGroup1F are all slave replicas, which are Partition1F, Partition4F and Partition7F respectively; the same is true for PartitionGroup2; in other words, each partition of the master partition group is a master replica, and each partition in the slave partition group is a slave replica, and the master-slave status of each partition in the partition group is consistent with the master-slave status of the partition group itself.

[0054] S104: If so, elect a primary partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy.

[0055] Exemplary, reference Figure 4 , in the case of Broker1 downtime, all partitions in Broker1 will fail. The primary replicas of Partition1, Partition4 and Partition7 are all in Broker1, and the primary replicas of Partition8 and Partition9 are also in Broker1. In the case of Broker1 downtime, Partition1L, Partition4L, Partition7L, Partition8L and Partition9L will all fail, so it is necessary to elect the primary replica for Partition1, Partition4, Partition7, Partition8 and Partition9. However, the primary replicas of Partition2, Partition3, Partition5 and Partition6 are not in Broker1. In the case of Broker1 downtime, Partition2F, Partition3F, Partition5F and Partition6F in Broker1 will all fail, but it is not necessary to elect the primary replica for Partition2, Partition3, Partition5 and Partition6.

[0056] Continue to refer Figure 4When the master replica election is performed for Partition1, Partition4, Partition7, Partition8 and Partition9, since the partition groups to which Partition1, Partition4 and Partition7 belong are all partition group 1, one of the three can be elected as the master partition group according to the preset partition group election strategy, among partition group 1 in Broker2, partition group 1 in Broker3 and partition group 1 in Broker4. For example, partition group 1 in Broker2 is selected as the elected master partition group, and partition group 1 in Broker3 and partition group 1 in Broker4 are both selected as slave partition groups.

[0057] Continue to refer Figure 4 Since the partition groups to which Partition8 and Partition9 belong are both partition group 2, one of the three can be elected as the primary partition group among partition group 2 in Broker2, partition group 2 in Broker3, and partition group 2 in Broker4 according to the preset partition group election strategy. For example, partition group 2 in Broker4 is elected as the primary partition group, and partition group 2 in Broker2 and partition group 2 in Broker3 are both elected as slave partition groups.

[0058] S106, write the partition group information of the elected primary partition group into Zookeeper, so as to instruct the Kafka controller through Zookeeper to update the partition group information of the primary partition group stored in the Kafka controller, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located, wherein the first change request is used to request the server node where the elected primary partition group is located, so that all slave copies of the elected primary partition group are changed to master copies.

[0059] For example, continue to refer to Figure 4, write the partition group information of partition group 1 in the elected Broker2 into Zookeeper, and write the partition group information of partition group 2 in the elected Broker4 into Zookeeper, and after writing, Zookeeper instructs the Kafka controller (Controller) to replace the partition group information of partition group 1 in Broker1 stored in the Kafka controller with the partition group information of partition group 1 in Broker2, and replace the partition group information of partition group 2 in Broker1 stored in the Kafka controller with the partition group information of partition group 2 in Broker4, and after the replacement, that is, after the update, the Kafka controller sends a first change request to Broker2, and the first change request is used to request Broker2 to change all the slave copies of partition group 1 in Broker2 to the master copy, and the Kafka controller sends another first change request to Broker4, and the first change request is used to request Broker4 to change all the slave copies of partition group 2 in Broker4 to the master copy.

[0060] The above is just an example. Ensure that the number of partition groups in the Broker is less than the number of partitions in the Broker, that is, the number of partitions in the partition group is multiple. It can be understood that the more partitions a partition group has, the greater the load reduction on Zookeeper.

[0061] The embodiment of the present application first implements the election of the primary partition group by dividing the partition group containing multiple partitions, and then directly obtains the elected primary replica according to the primary partition group. During the election, it is only necessary to elect according to the partition group. Compared with the traditional technology of electing the primary replica according to a single partition, the ZooKeeper load is greatly reduced, the network congestion is improved, and the election efficiency is improved.

[0062] In an exemplary embodiment, referring to Figure 5 Before selecting a primary partition group from the partition groups in multiple server nodes according to a preset partition group election strategy in step S104, the method for electing a primary replica of a Kafka partition may further include the following steps S502 to S508.

[0063] S502, determining whether all slave replicas of the partition group in the plurality of server nodes are valid.

[0064] For example, continue to refer to Figure 4, determine whether all the slave replicas of partition group 1 in Broker2, all the slave replicas of partition group 1 in Broker3, and all the slave replicas of partition group 1 in Broker4 are valid; and, determine whether all the slave replicas of partition group 2 in Broker2, all the slave replicas of partition group 2 in Broker3, and all the slave replicas of partition group 2 in Broker4 are valid.

[0065] S504, if all slave copies of the partition group in each server node are valid, then executing the step of selecting a master partition group from the partition groups in multiple server nodes according to a preset partition group election strategy.

[0066] For example, continue to refer to Figure 4 , if all the slave replicas of partition group 1 in Broker2, all the slave replicas of partition group 1 in Broker3, and all the slave replicas of partition group 1 in Broker4 are valid, then execute step S104, and perform master replica election for Partition1, Partition4, and Partition7 in partition group 1 in Broker2, partition group 1 in Broker3, and partition group 1 in Broker4; and, if all the slave replicas of partition group 2 in Broker2, all the slave replicas of partition group 2 in Broker3, and all the slave replicas of partition group 2 in Broker4 are valid, then execute step S104, and perform master replica election for Partition8 and Partition9 in partition group 2 in Broker2, partition group 2 in Broker3, and partition group 2 in Broker4.

[0067] S506: If at least one slave replica is invalid, the server node where the invalid slave replica is located is removed from the multiple server nodes, and after the removal, a master partition group is elected from the partition groups in the remaining server nodes according to a preset partition group election strategy.

[0068] For example, continue to refer to Figure 4, if at least one slave replica of partition group 1 in Broker3 is invalid, then in partition group 1 in Broker2 and partition group 1 in Broker4, primary replica election is performed for Partition1, Partition4 and Partition7, and Broker3 is eliminated. If at least one slave replica of partition group 2 in Broker2 is invalid, and at least one slave replica of partition group 2 in Broker3 is invalid, then in partition group 2 in Broker4, primary replica election is performed for Partition8 and Partition9, and Broker2 and Broker3 are eliminated. In fact, since only partition group 2 in Broker4 is left, partition group 2 in Broker4 can be directly used as the primary replica group for Partition8 and Partition9.

[0069] S508: If there is at least one invalid slave replica in the partition group in each server node, the election is terminated.

[0070] For example, continue to refer to Figure 4 , if at least one slave replica of partition group 1 in Broker2, at least one slave replica of partition group 1 in Broker3, and at least one slave replica of partition group 1 in Broker4 are all invalid, then the election ends and the election of the primary replicas of Partition1, Partition4, and Partition7 is withdrawn. If at least one slave replica of partition group 2 in Broker2, at least one slave replica of partition group 2 in Broker3, and at least one slave replica of partition group 2 in Broker4 are all invalid, then the election ends and the election of the primary replicas of Partition8 and Partition9 is withdrawn.

[0071] In an exemplary embodiment, the method for electing a primary replica of a Kafka partition may further include: when Zookeeper detects that at least one server node among the multiple server nodes is down, determining that a partition primary replica election is required.

[0072] In an exemplary embodiment, referring to Figure 6 The method for electing the primary replica of a Kafka partition may also include the following steps S108~S110.

[0073] S108: If the partition that needs to perform master replica election does not have a subordinate partition group, a master replica is elected from the slave replicas in multiple server nodes according to a preset partition election strategy.

[0074] For example, continue to refer to Figure 3, Partition2, Partition3, Partition5, and Partition6 do not belong to any partition group, that is, they are not added to the partition group. The primary replicas of Partition2 and Partition5 are both in Broker2. Therefore, if Broker2 goes down, the primary replica of Partition2 can be elected from Partition2F in Broker1, Partition2F in Broker3, and Partition2F in Broker4. Similarly, if Broker2 goes down, the primary replica of Partition5 can be elected from Partition5F in Broker1, Partition5F in Broker3, and Partition5F in Broker4. In the case of Broker2 going down, since the primary replicas of partitions other than Partition2 and Partition5 are not in Broker2, there is no need to elect primary replicas for partitions other than Partition2 and Partition5.

[0075] S110, write the partition information of the elected primary replica into Zookeeper, so as to instruct the Kafka controller through Zookeeper to update the partition information of the primary replica stored in the Kafka controller, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located, wherein the second change request is used to request the server node where the elected primary replica is located, so as to realize the change of the elected primary replica from a slave replica to a master replica.

[0076] For example, continue to refer to Figure 3 , write the partition information of the elected primary replica of Partition2 into Zookeeper, and write the partition information of the elected primary replica of Partition5 into Zookeeper, and after writing, Zookeeper instructs the Kafka controller to replace the partition information of Partition2 in Broker2 stored in the Kafka controller with the partition information of the elected primary replica of Partition2, and replace the partition information of Partition5 in Broker2 stored in the Kafka controller with the partition information of the elected primary replica of Partition5, and after the replacement, that is, after the update, the Kafka controller sends a second change request to the Broker where the elected primary replica of Partition2 is located, and the second change request is used to request the Broker to implement the change of the elected primary replica of Partition2 from a slave replica to a primary replica, and the same is true for Partition5.

[0077] In an exemplary embodiment, when the partition that needs to perform master replica election has a partition group to which it belongs, among the multiple server nodes, the partition group in each server node has a partition group level listener registered in Zookeeper; when the partition that needs to perform master replica election does not have a partition group to which it belongs, among the multiple server nodes, the slave replica in each server node has a partition level listener registered in Zookeeper.

[0078] Since the present application uses each partition group level listener to monitor each partition group separately, compared with the traditional technology that uses each partition level listener to monitor each partition, if one listener is used as a load of Zookeeper, this greatly reduces the load of Zookeeper, improves network congestion, and improves election efficiency.

[0079] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0080] Based on the same inventive concept, the embodiment of the present application also provides a Kafka partition master replica election device for implementing the above-mentioned Kafka partition master replica election method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the election device for one or more Kafka partition master replicas provided below can be referred to the limitations of the Kafka partition master replica election method above, and will not be repeated here.

[0081] In an exemplary embodiment, Figure 7 As shown, a device for electing a primary replica of a Kafka partition is provided, including:

[0082] The group election module 710 is used to determine whether the partition that needs to perform the primary replica election has a subordinate partition group when it is determined that the partition primary replica election needs to be performed. If so, the primary partition group is selected from the partition groups in the multiple server nodes according to the preset partition group election strategy;

[0083] An update change module 720 is used to write the partition group information of the elected primary partition group into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located;

[0084] The first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

[0085] In an exemplary embodiment, the group election module 710 is also used to determine whether all slave copies of the partition groups in the multiple server nodes are valid before selecting the master partition group from the partition groups in the multiple server nodes according to the preset partition group election strategy; if all slave copies of the partition groups in the server nodes are valid, then executing the step of selecting the master partition group from the partition groups in the multiple server nodes according to the preset partition group election strategy.

[0086] In an exemplary embodiment, the group election module 710 is also used to, if at least one slave replica is invalid, remove the server node where the invalid slave replica is located from the multiple server nodes, and after the removal, elect the master partition group from the partition groups in the remaining server nodes according to a preset partition group election strategy; if there is at least one invalid slave replica in the partition groups in each server node, the election is terminated.

[0087] In an exemplary embodiment, the group election module 710 is further configured to determine that partition master replica election is required when Zookeeper detects that at least one server node among the multiple server nodes is down.

[0088] In an exemplary embodiment, the group election module 710 is also used to elect a master replica from slave replicas in multiple server nodes according to a preset partition election strategy if the partition that needs to perform master replica election does not have a subordinate partition group.

[0089] In an exemplary embodiment, the update change module 720 is also used to write the partition information of the elected primary replica into Zookeeper, so as to instruct the Kafka controller through Zookeeper to update the partition information of the primary replica stored in the Kafka controller, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located; wherein the second change request is used to request the server node where the elected primary replica is located to realize the change of the elected primary replica from a slave replica to a master replica.

[0090] In an exemplary embodiment, when the partition that needs to perform master replica election exists to which it belongs, among the multiple server nodes, the partition group in each server node has a partition group level listener registered in Zookeeper; when the partition that needs to perform master replica election does not exist to which it belongs, among the multiple server nodes, the slave replica in each server node has a partition level listener registered in Zookeeper.

[0091] The above-mentioned Kafka partition master replica election device, each module in the device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0092] In an exemplary embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the diagnostic instrument is used to provide computing and control capabilities. The memory of the diagnostic instrument includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the diagnostic instrument is used to store N code data to be configured. The input / output interface of the diagnostic instrument is used to exchange information between the processor and an external device. The communication interface of the diagnostic instrument is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps of any of the above method embodiments are implemented.

[0093] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned embodiments of the method for electing the primary replica of a Kafka partition are implemented.

[0094] In an exemplary embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned embodiments of the method for electing a primary replica of a Kafka partition.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0096] In the description of this specification, the description with reference to the terms "some embodiments", "other embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic description of the above terms does not necessarily refer to the same embodiment or example.

[0097] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for electing a primary replica of a Kafka partition, characterized in that: The method comprises: When it is determined that a partition master replica election is required, determine whether the partition that requires a master replica election has a subordinate partition group; If so, a primary partition group is selected from the partition groups in the multiple server nodes according to a preset partition group election strategy; The partition group information of the elected primary partition group is written into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located; The first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

2. The method according to claim 1, characterized in that Before selecting a primary partition group from the partition groups in the plurality of server nodes according to a preset partition group election strategy, the method further includes: Determine whether all slave copies of the partition group in multiple server nodes are valid; If all slave copies of the partition group in each server node are valid, a step of selecting a master partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy is performed.

3. The method according to claim 2, characterized in that The method further comprises: If at least one slave replica is invalid, the server node where the invalid slave replica is located is removed from the multiple server nodes, and after the removal, a master partition group is elected from the partition groups in the remaining server nodes according to a preset partition group election strategy; If there is at least one invalid slave replica in the partition group in each server node, the election is terminated.

4. The method according to claim 1, characterized in that The method further comprises: When Zookeeper detects that at least one of the multiple server nodes is down, it is determined that partition master replica election is required.

5. The method according to claim 1, characterized in that The method further comprises: If the partition that needs to elect a master replica does not belong to any partition group, the master replica is elected from the slave replicas in multiple server nodes according to the preset partition election strategy; The partition information of the elected primary replica is written into Zookeeper, so as to instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located; The second change request is used to request the server node where the elected master copy is located, so as to change the elected master copy from a slave copy to a master copy.

6. The method according to claim 5, characterized in that When the partition to be elected as the primary replica has a partition group to which it belongs, in the plurality of server nodes, the partition group in each server node has a partition group level listener registered in Zookeeper; When the partition that needs to perform master replica election does not have a partition group to which it belongs, among the multiple server nodes, the slave replica in each server node registers a partition-level listener in Zookeeper.

7. A Kafka partition master replica election device, characterized in that: The device comprises: A group election module is used to determine whether the partition that needs to perform the primary replica election has a subordinate partition group when it is determined that the partition primary replica election needs to be performed. If so, a primary partition group is selected from the partition groups in the multiple server nodes according to a preset partition group election strategy; An update change module, used for writing the partition group information of the elected primary partition group into Zookeeper, so as to instruct the Kafka controller through Zookeeper to update the partition group information of the primary partition group stored in the Kafka controller, and after the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located; The first change request is used to request the server node where the elected master partition group is located, so that all slave copies of the elected master partition group are changed to master copies.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images