Method and device for electing master replica of kafka partition, equipment, medium and product

Abstract

The application relates to a Kafka partition master replica election method and device, equipment, a medium and a product. The method comprises the following steps: in the case that it is determined that partition master replica election is needed, judging whether a partition needing master replica election exists in a subordinate partition group; if yes, according to a preset partition group election strategy, a master partition group is elected from the partition group in a plurality of server nodes; partition group information of the elected master partition group is written into Zookeeper, so as to instruct a Kafka controller to update the partition group information of the master partition group stored in the Kafka controller through the Zookeeper, and after the update, the Kafka controller sends a first change request to a server node where the elected master partition group is located; wherein the first change request is used for requesting the server node where the elected master partition group is located to change each slave replica of the elected master partition group into a master replica; in this way, the problem of excessive ZooKeeper load and network congestion caused by partition master replica election is solved.

Description

Technical Field

[0001] This application relates to the field of Kafka cluster technology, and in particular to a method, apparatus, device, medium and product for electing a primary replica of a Kafka partition. Background Technology

[0002] Kafka is a distributed message streaming platform that supports multiple partitions, multiple replicas, and coordination based on Zookeeper. It is also an open-source message engine system based on the publish-subscribe pattern.

[0003] Kafka partition leader election typically occurs when the leader of a partition fails. That is, when the leader of one or more partitions in a Kafka cluster fails, a new leader needs to be elected from the partition's followers.

[0004] In traditional technologies, when Kafka has more than 10,000 partitions, the election of the primary replica of the partitions can lead to problems such as excessive load on Zookeeper and network congestion. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, device, medium, and product for electing Kafka partition master replicas that can solve the problems of excessive ZooKeeper load and network congestion caused by partition master replica election.

[0006] In a first aspect, embodiments of this application provide a method for electing a primary replica of a Kafka partition, the method comprising:

[0007] If it is determined that a partition master copy election is required, determine whether the partition to which the master copy election is required belongs to a partition group;

[0008] If it exists, a primary partition group is elected from the partition groups of multiple server nodes according to the preset partition group election strategy.

[0009] Write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.

[0010] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.

[0011] In one embodiment, before electing a primary partition group from the partition groups among multiple server nodes according to a preset partition group election strategy, the method further includes:

[0012] Determine whether each replica of the partition group in the multiple server nodes is valid;

[0013] If all slave replicas of the partition group in each server node are valid, then the step of electing a primary partition group from the partition groups in multiple server nodes according to a preset partition group election strategy is executed.

[0014] In one embodiment, the method further includes:

[0015] If at least one replica is invalid, the server node containing the invalid replica is removed from the plurality of server nodes, and after removal, a primary partition group is elected from the partition groups of the remaining server nodes according to the preset partition group election strategy.

[0016] If each partition group in each server node has at least one invalid slave replica, the election ends.

[0017] In one embodiment, the method further includes:

[0018] If Zookeeper detects that at least one of the multiple server nodes has crashed, it determines that a partition master replica election is required.

[0019] In one embodiment, the method further includes:

[0020] If the partition that needs to elect a primary replica does not belong to a partition group, a primary replica will be elected from the replicas of multiple server nodes according to the preset partition election strategy.

[0021] The partition information of the elected primary replica is written to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller. After the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located.

[0022] The second change request is used to request the server node where the elected primary replica is located to change the elected primary replica from a secondary replica to a primary replica.

[0023] In one embodiment, if the partition that needs to perform master replica election has a partition group to which it belongs, then in the plurality of server nodes, each of the partition groups in the server node registers a partition group-level listener in Zookeeper;

[0024] If the partition requiring master replica election does not belong to a partition group, then among the multiple server nodes, each server node's slave replica registers a partition-level listener in ZooKeeper.

[0025] Secondly, embodiments of this application provide a Kafka partition primary replica election device, the device comprising:

[0026] The group election module is used to determine whether the partition to be elected as the primary replica needs to be elected has a corresponding partition group when it is determined that a primary replica election needs to be performed. If so, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy.

[0027] The update and change module is used to write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.

[0028] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.

[0029] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0031] Fifthly, embodiments of this application provide a computer program product, including a computer program, characterized in that the computer program, when executed by a processor, implements the method described in the first aspect.

[0032] The aforementioned method, apparatus, device, medium, and product for electing a primary replica of a Kafka partition, upon determining that a primary replica election is required, firstly determines whether the partition requiring primary replica election belongs to a partition group. If the partition requiring primary replica election belongs to a partition group, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy. Then, the partition group information of the elected primary partition group is written to ZooKeeper, so that ZooKeeper instructs the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends a first change request to the server node where the elected primary partition group resides, wherein the first change request is used to request the server node where the elected primary partition group resides to change all slave replicas of the elected primary partition group to primary replicas.

[0033] Thus, in this embodiment, the primary partition group is first elected by dividing the partitions into partition groups containing multiple partitions, and then the elected primary replica is directly obtained based on the primary partition group. During the election, it is only necessary to elect according to the partition group. Compared with the traditional technology of electing the primary replica according to a single partition, this greatly reduces the ZooKeeper load, improves network congestion, and increases election efficiency. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a method for electing the primary replica of a Kafka partition, as shown in one embodiment.

[0036] Figure 2 This is one of the schematic diagrams of a Kafka cluster as an example;

[0037] Figure 3 This is the second schematic diagram of a Kafka cluster in one embodiment;

[0038] Figure 4 This is the third schematic diagram of a Kafka cluster in one embodiment;

[0039] Figure 5 One of the flowcharts for the election of the primary replica of a Kafka partition, as an example of an embodiment, is also included.

[0040] Figure 6The second flowchart of the Kafka partition primary replica election method, which is one embodiment, is also included;

[0041] Figure 7 This is a block diagram of a Kafka partition primary replica election device according to an embodiment. Detailed Implementation

[0042] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0044] It is understood that the terms "first," "second," etc., used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.

[0045] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0046] It is understandable that "at least one" refers to one or more, and "multiple" refers to two or more. "At least a part of an element" refers to part or all of an element.

[0047] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0048] In one example embodiment, reference is made to Figure 1This paper provides a method for electing the primary replica of a Kafka partition, which may include the following steps S102~S106.

[0049] S102, if it is determined that a partition primary replica election is required, determine whether the partition to which the primary replica election is required has a corresponding partition group.

[0050] A Kafka cluster can include multiple topics, each topic can include multiple partitions, and these partitions can be distributed across different server nodes (Brokers); each partition can include multiple replicas.

[0051] For example, refer to Figure 2 In a Kafka cluster, a specific topic may consist of nine partitions, from Partition1 to Partition9. Each of these nine partitions comprises four replicas, distributed across Broker1 through Broker4. Taking Partition1 as an example: Partition1 has four replicas distributed across Broker1 through Broker4. The four replicas of Partition1 include Partition1 itself; in other words, Partition1 itself is considered one of the four replicas of Partition1. Partition1L is the elected primary replica among the four replicas of Partition1, and Partition1F is the secondary replica among the four replicas of Partition1.

[0052] For example, refer to Figure 3 In Broker1 to Broker4, Partition1, Partition4 and Partition7 all belong to a partition group, and the partition group they belong to is PartitionGroup1; Partition8 and Partition9 both belong to a partition group, and the partition group they belong to is PartitionGroup2; Partition2, Partition3, Partition5 and Partition6 do not belong to any partition group, that is, they are not part of any partition group.

[0053] Continue to refer to Figure 3As can be seen, for PartitionGroup1, PartitionGroup1 exists in Broker1 to Broker4. When PartitionGroup1 in Broker1 is the primary partition group (PartitionGroup1L) and PartitionGroup1 in Broker2 to Broker4 is the secondary partition group (PartitionGroup1F), the partitions of PartitionGroup1L are all primary replicas, namely Partition1L, Partition4L, and Partition7L, and the partitions of PartitionGroup1F are all secondary replicas, namely Partition1F, Partition4F, and Partition7F. The same applies to PartitionGroup2. In other words, each partition in the primary partition group is a primary replica, and each partition in the secondary partition group is a secondary replica. The primary / secondary status of each partition in the partition group is consistent with the primary / secondary status of the partition group itself.

[0054] S104, if it exists, then according to the preset partition group election strategy, a primary partition group is elected from the partition groups in the multiple server nodes.

[0055] For example, refer to Figure 4 If Broker1 fails, all partitions within Broker1 will become unavailable. The primary replicas of Partition1, Partition4, and Partition7 are all on Broker1, as are the primary replicas of Partition8 and Partition9. If Broker1 fails, Partition1L, Partition4L, Partition7L, Partition8L, and Partition9L will also become unavailable. Therefore, a primary replica election is required for Partition1, Partition4, Partition7, Partition8, and Partition9. However, the primary replicas of Partition2, Partition3, Partition5, and Partition6 are not on Broker1. If Broker1 fails, Partition2F, Partition3F, Partition5F, and Partition6F within Broker1 will become unavailable, but a primary replica election is not required for Partition2, Partition3, Partition5, and Partition6.

[0056] Continue to refer to Figure 4When performing primary replica elections for Partition1, Partition4, Partition7, Partition8, and Partition9, since Partition1, Partition4, and Partition7 all belong to Partition Group 1, one of the three can be elected as the primary partition group according to the preset partition group election strategy: Partition Group 1 in Broker2, Partition Group 1 in Broker3, and Partition Group 1 in Broker4. For example, Partition Group 1 in Broker2 can be elected as the primary partition group, while Partition Group 1 in Broker3 and Partition Group 1 in Broker4 can both be elected as secondary partition groups.

[0057] Continue to refer to Figure 4 Since Partition8 and Partition9 both belong to partition group 2, one of the three partition groups in Broker2, Broker3, and Broker4 can be elected as the primary partition group according to the preset partition group election strategy. For example, partition group 2 in Broker4 can be elected as the primary partition group, while partition groups 2 in Broker2 and Broker3 can both be elected as secondary partition groups.

[0058] S106, write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located. The first change request is used to request the server node where the elected primary partition group is located to change each replica of the elected primary partition group to a primary replica.

[0059] For example, continue to refer to Figure 4The process involves writing the partition group information of partition group 1 in the elected Broker2 and partition group 2 in the elected Broker4 into ZooKeeper. After writing, ZooKeeper instructs the Kafka Controller to replace the partition group information of partition group 1 in Broker1 stored in the Kafka Controller with the partition group information of partition group 1 in Broker2, and to replace the partition group information of partition group 2 in Broker1 stored in the Kafka Controller with the partition group information of partition group 2 in Broker4. After the replacement, i.e., after the update, the Kafka Controller sends a first change request to Broker2, which requests Broker2 to change all slave replicas of partition group 1 in Broker2 to primary replicas. The Kafka Controller also sends another first change request to Broker4, which requests Broker4 to change all slave replicas of partition group 2 in Broker4 to primary replicas.

[0060] The above is just an example. Make sure the number of partition groups in the Broker is less than the number of partitions in the Broker. In other words, the number of partitions in a partition group should be multiple. It can be understood that the more partitions in a partition group, the greater the reduction in load on Zookeeper.

[0061] This application's embodiment first elects a primary partition group by dividing the system into partition groups containing multiple partitions, and then directly obtains the elected primary replica based on the primary partition group. During the election, only the partition group needs to be selected, which significantly reduces ZooKeeper's load compared to traditional techniques that elect a primary replica based on a single partition. This also improves network congestion and election efficiency.

[0062] In one exemplary embodiment, reference is made to Figure 5 Before electing the primary partition group from the partition groups among multiple server nodes according to the preset partition group election strategy in step S104, the Kafka partition primary replica election method may further include the following steps S502 to S508.

[0063] S502, determine whether each replica of the partition group in the multiple server nodes is valid.

[0064] For example, continue to refer to Figure 4Determine whether each slave copy of partition group 1 in Broker2, each slave copy of partition group 1 in Broker3, and each slave copy of partition group 1 in Broker4 are valid; and determine whether each slave copy of partition group 2 in Broker2, each slave copy of partition group 2 in Broker3, and each slave copy of partition group 2 in Broker4 are valid.

[0065] S504, if all slave replicas of the partition group in each server node are valid, then the step of electing a master partition group from the partition groups in multiple server nodes according to a preset partition group election strategy is executed.

[0066] For example, continue to refer to Figure 4 If all slave replicas of partition group 1 in Broker2, partition group 1 in Broker3, and partition group 1 in Broker4 are valid, then step S104 is executed, in which primary replica election is performed for Partition1, Partition4, and Partition7 in partition group 1 of Broker2, partition group 1 of Broker3, and partition group 1 of Broker4; and if all slave replicas of partition group 2 in Broker2, partition group 2 in Broker3, and partition group 2 in Broker4 are valid, then step S104 is executed, in which primary replica election is performed for Partition8 and Partition9 in partition group 2 of Broker2, partition group 2 of Broker3, and partition group 2 of Broker4.

[0067] S506, if at least one slave replica is invalid, then remove the server node containing the invalid slave replica from the plurality of server nodes, and after removal, elect a primary partition group from the partition groups of the remaining server nodes according to the preset partition group election strategy.

[0068] For example, continue to refer to Figure 4If at least one slave replica of partition group 1 in Broker3 is invalid, then in partition group 1 of Broker2 and partition group 1 of Broker4, a primary replica election is performed for Partition1, Partition4, and Partition7, in which Broker3 is removed. If at least one slave replica of partition group 2 in Broker2 and at least one slave replica of partition group 2 in Broker3 are invalid, then in partition group 2 of Broker4, a primary replica election is performed for Partition8 and Partition9, in which Broker2 and Broker3 are removed. In fact, since only partition group 2 in Broker4 remains at this point, partition group 2 in Broker4 can be directly used as the primary replica group for Partition8 and Partition9.

[0069] S508, if each partition group in each server node has at least one invalid slave replica, then the election ends.

[0070] For example, continue to refer to Figure 4 If at least one slave copy of partition group 1 in Broker2, at least one slave copy of partition group 1 in Broker3, and at least one slave copy of partition group 1 in Broker4 are all invalid, then the election ends, and the election of primary copies for Partition1, Partition4, and Partition7 is withdrawn. If at least one slave copy of partition group 2 in Broker2, at least one slave copy of partition group 2 in Broker3, and at least one slave copy of partition group 2 in Broker4 are all invalid, then the election ends, and the election of primary copies for Partition8 and Partition9 is withdrawn.

[0071] In an exemplary embodiment, the Kafka partition primary replica election method may further include: determining that a partition primary replica election is required when Zookeeper detects that at least one of the plurality of server nodes has crashed.

[0072] In one exemplary embodiment, reference is made to Figure 6 The election method for the primary replica of a Kafka partition may also include the following steps S108~S110.

[0073] S108 If the partition that needs to elect a primary replica does not belong to a partition group, then a primary replica is elected from the replicas of multiple server nodes according to the preset partition election strategy.

[0074] For example, continue to refer to Figure 3Partitions 2, 3, 5, and 6 do not belong to any partition groups, meaning they are not part of any partition group. The primary replicas of Partitions 2 and 5 reside on Broker 2. Therefore, if Broker 2 fails, the primary replica of Partition 2 can be elected from among Partitions 2F in Broker 1, 3, and 4. Similarly, if Broker 2 fails, the primary replica of Partition 5 can be elected from among Partitions 5F in Broker 1, 3, and 4. Since the primary replicas of partitions other than Partitions 2 and 5 are not on Broker 2, there is no need to elect a primary replica for these partitions if Broker 2 fails.

[0075] S110: Write the partition information of the elected primary replica to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller. After the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located. The second change request is used to request the server node where the elected primary replica is located to change the elected primary replica from a secondary replica to a primary replica.

[0076] For example, continue to refer to Figure 3 The partition information of the primary replica of the elected Partition2 is written to ZooKeeper, as is the partition information of the primary replica of the elected Partition5. After writing, ZooKeeper instructs the Kafka controller to replace the partition information of Partition2 in Broker2 stored in the Kafka controller with the partition information of the primary replica of the elected Partition2, and to replace the partition information of Partition5 in Broker2 stored in the Kafka controller with the partition information of the primary replica of the elected Partition5. After the replacement, i.e., after the update, the Kafka controller sends a second change request to the Broker where the primary replica of the elected Partition2 is located. This second change request is used to request the Broker to complete the change from a replica to a primary replica of the elected Partition2. The same applies to Partition5.

[0077] In one exemplary embodiment, if the partition to which master replica election is required has a partition group to which it belongs, then in the plurality of server nodes, each server node has a partition group-level listener registered in ZooKeeper for the partition group; if the partition to which master replica election is required does not have a partition group to which it belongs, then in the plurality of server nodes, each server node has a slave replica registered in ZooKeeper for a partition-level listener.

[0078] Since this application uses listeners at each partition group level to listen to each partition group separately, compared to the traditional technology where each partition-level listener listens to each partition, and each listener acts as a load on Zookeeper, this greatly reduces the load on Zookeeper, improves network congestion, and increases election efficiency.

[0079] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0080] Based on the same inventive concept, this application also provides a Kafka partition primary replica election apparatus for implementing the Kafka partition primary replica election method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more Kafka partition primary replica election apparatus embodiments provided below can be found in the limitations of the Kafka partition primary replica election method described above, and will not be repeated here.

[0081] In one exemplary embodiment, such as Figure 7 As shown, a mechanism for electing primary replicas of Kafka partitions is provided, including:

[0082] The group election module 710 is used to determine whether the partition to be elected as the primary replica needs to be elected has a corresponding partition group when it is determined that a primary replica election needs to be performed. If it does, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy.

[0083] The update and change module 720 is used to write the partition group information of the elected primary partition group to Zookeeper, so as to instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller through Zookeeper. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.

[0084] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.

[0085] In an exemplary embodiment, the group election module 710 is further configured to determine whether each replica of the partition group in the multiple server nodes is valid before electing a primary partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy; if each replica of the partition group in the multiple server nodes is valid, then the step of electing a primary partition group from the partition groups in the multiple server nodes according to the preset partition group election strategy is executed.

[0086] In an exemplary embodiment, the group election module 710 is further configured to, if at least one slave replica is invalid, remove the server node containing the invalid slave replica from the plurality of server nodes, and after removal, elect a primary partition group from the partition groups in the remaining server nodes according to a preset partition group election strategy; if each partition group in each server node has at least one invalid slave replica, then the election ends.

[0087] In an exemplary embodiment, the group election module 710 is further configured to determine that a partition master replica election is required when Zookeeper detects that at least one of the plurality of server nodes has crashed.

[0088] In an exemplary embodiment, the group election module 710 is further configured to, if the partition to which the primary replica election is required does not belong to a partition group, elect a primary replica from the secondary replicas among multiple server nodes according to a preset partition election strategy.

[0089] In an exemplary embodiment, the update / change module 720 is further configured to write the partition information of the elected primary replica into Zookeeper, so as to instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located; wherein, the second change request is used to request the server node where the elected primary replica is located to realize the change of the elected primary replica from a secondary replica to a primary replica.

[0090] In one exemplary embodiment, if the partition to which master replica election is required has a partition group to which it belongs, then in the plurality of server nodes, each server node registers a partition group-level listener for the partition group in ZooKeeper; if the partition to which master replica election is required does not have a partition group to which it belongs, then in the plurality of server nodes, each server node's slave replica registers a partition-level listener in ZooKeeper.

[0091] The aforementioned Kafka partition primary replica election mechanism can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0092] In an exemplary embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the diagnostic instrument provides computing and control capabilities. The memory of the diagnostic instrument includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the diagnostic instrument stores N configuration code data. The I / O interfaces of the diagnostic instrument are used for exchanging information between the processor and external devices. The communication interface of the diagnostic instrument is used for communicating with an external terminal via a network connection. When the computer program is executed by the processor, it implements the steps of any of the above method embodiments.

[0093] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described embodiments of the method for electing a primary replica of any Kafka partition.

[0094] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described embodiments of the method for electing a primary replica of any Kafka partition.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0096] In the description of this specification, references to terms such as "some embodiments," "other embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for electing a Kafka partition primary replica, characterized in that, The method comprises: In the case of determining that partition master replica election is needed, it is judged whether the partition needing master replica election exists a subordinate partition group; If yes, a master partition group is elected from the partition group in the plurality of server nodes according to a preset partition group election strategy; The partition group information of the elected master partition group is written into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the master partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected master partition group is located; The first change request is used to request the server node where the elected master partition group is located to change each slave replica of the elected master partition group into a master replica.

2. The method of claim 1, wherein, Before the master partition group is elected from the partition group in the plurality of server nodes according to the preset partition group election strategy, the method further comprises: It is judged whether each slave replica of the partition group in the plurality of server nodes is valid; If each slave replica of the partition group in each server node is valid, the step of electing a master partition group from the partition group in the plurality of server nodes according to the preset partition group election strategy is executed.

3. The method of claim 2, wherein, The method further comprises: If at least one slave replica is invalid, the server node where the invalid slave replica is located is excluded from the plurality of server nodes, and after the exclusion, a master partition group is elected from the partition group in the remaining server nodes according to the preset partition group election strategy; If each server node has at least one invalid slave replica of the partition group, the election is ended.

4. The method of claim 1, wherein, The method further comprises: In the case that Zookeeper listens to the occurrence of at least one server node in the plurality of server nodes being down, it is determined that partition master replica election is needed.

5. The method of claim 1, wherein, The method further comprises: If the partition needing master replica election does not exist a subordinate partition group, a master replica is elected from the slave replicas in the plurality of server nodes according to a preset partition election strategy; The partition information of the elected master replica is written into Zookeeper, so as to instruct the Kafka controller to update the partition information of the master replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected master replica is located; The second change request is used to request the server node where the elected master replica is located to change the elected master replica from a slave replica to a master replica.

6. The method of claim 5, wherein, In the case that the partition needing master replica election exists a subordinate partition group, in the plurality of server nodes, the partition group in each server node is registered with a partition group level listener in Zookeeper; In the case that the partition needing master replica election does not exist a subordinate partition group, in the plurality of server nodes, each slave replica in each server node is registered with a partition level listener in Zookeeper.

7. A mechanism for electing a primary replica in a Kafka partition, characterized in that, The device comprises: The group election module is used to determine whether the partition to be elected as the primary replica needs to be elected has a corresponding partition group when it is determined that a primary replica election needs to be performed. If so, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy. The update and change module is used to write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located. The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. When the processor executes the computer program, it implements the method according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images