A cluster switching method, apparatus, electronic device and storage medium

Abstract

This application provides a cluster switching method, apparatus, electronic device, and storage medium, relating to the field of communication technology, to solve the problems of untimely or lost data recovery and the inability to guarantee the real-time performance and reliability of a fault center system. The method includes: obtaining the current working status of the primary cluster and the standby cluster in the Kafka cluster; when the primary cluster is unavailable and the standby cluster is available, obtaining a first offset, a second offset, and a third offset; determining a target offset based on the first offset, the second offset, and the third offset; and sending the target offset to the standby cluster so that the standby cluster processes consumed data according to the target offset.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a trunking handover method, apparatus, electronic device, and storage medium. Background Technology

[0002] As one of the 10 capability centers of the network middleware platform, the fault center system is responsible for processing fault data nationwide and promptly identifying equipment and network problems. It occupies an important position as the data hub of the entire fault platform system. Therefore, in addition to having the function of real-time alarm processing, the fault center system also needs to adopt a protection mechanism to ensure the reliability of core business components.

[0003] Currently, fault center systems typically use a distributed messaging system (Kafka cluster) as the core business component. Summary of the Invention

[0004] This application provides a cluster switching method, apparatus, electronic device, and storage medium, relating to the field of communication technology. By calculating the target offset, it ensures smooth service switching and can solve the problems of untimely or lost data recovery and the inability to guarantee the real-time performance and reliability of the fault center system.

[0005] Firstly, this application provides a cluster switching method, comprising: obtaining the current working status of the primary cluster and the standby cluster in the Kafka cluster; when the working status of the primary cluster is unavailable and the working status of the standby cluster is available, obtaining a first offset, a second offset, and a third offset; the first offset is the maximum offset of the target consumer group in the primary cluster; the second offset is the offset currently consumed by the target consumer group in the primary cluster; the third offset is the maximum offset of the target consumer group in the standby cluster; determining a target offset based on the first offset, the second offset, and the third offset; and sending the target offset to the standby cluster so that the standby cluster processes the consumed data according to the target offset.

[0006] The technical solution provided in this application has at least the following beneficial effects: by analyzing the working status of the primary cluster and the standby cluster in the Kafka cluster and calculating the target offset, the execution of the cluster switching work is determined. Compared with the existing cluster switching methods, it can ensure the smoothness of business switching, thereby ensuring the real-time performance and reliability of the fault center system.

[0007] In one possible implementation, determining the target offset based on the first offset, the second offset, and the third offset includes: determining the fourth offset of the target consumer group currently not consumed in the main cluster based on the first offset and the second offset; and determining the target offset based on the third offset and the fourth offset.

[0008] In another possible implementation, the target offset is determined based on the third offset and the fourth offset, including: using the difference between the third offset and the fourth offset as the target offset.

[0009] In another possible implementation, the target offset is determined based on the third offset and the fourth offset, including: obtaining the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; determining the compensation value based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; using the sum of the first target offset and the compensation value as the target offset; the first target offset is the difference between the third offset and the fourth offset.

[0010] In another possible implementation, the compensation value is determined based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment. This includes using the product of the data traffic and the first value as the compensation value; the first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster.

[0011] In another possible implementation, the working states of the primary cluster and the standby cluster include available and unavailable, respectively. The method also includes issuing an alarm when both the primary cluster and the standby cluster are unavailable, or when the primary cluster is available and the standby cluster is unavailable.

[0012] Secondly, this application provides a cluster switching device, which includes an acquisition module and a processing module.

[0013] The acquisition module is used to obtain the working status of the primary cluster and the standby cluster in the Kafka cluster at the current moment; when the working status of the primary cluster is unavailable and the working status of the standby cluster is available, it obtains the first offset, the second offset, and the third offset.

[0014] The processing module is used to determine the target offset based on the first offset, the second offset, and the third offset; and send the target offset to the standby cluster so that the standby cluster processes the consumed data according to the target offset.

[0015] Optionally, the acquisition module is specifically used to acquire data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment.

[0016] Optionally, the processing module is specifically used to determine the fourth offset of the target consumer group that is not currently consumed in the main cluster based on the first offset and the second offset; and to determine the target offset based on the third offset and the fourth offset.

[0017] Optionally, the processing module is specifically used to take the difference between the third offset and the fourth offset as the target offset.

[0018] Optionally, the processing module is also used to determine the compensation value based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; the sum of the first target offset and the compensation value is used as the target offset; the first target offset is the difference between the third offset and the fourth offset.

[0019] Optionally, the processing module is also used to use the product of the data flow and a first value as a compensation value; the first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster.

[0020] Optionally, the processing module is also configured to issue an alarm when both the primary cluster and the standby cluster are unavailable, or when the primary cluster is available but the standby cluster is unavailable.

[0021] Thirdly, this application provides an electronic device comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to perform a cluster switching method as described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of a server, enables the server to perform the method provided by the first aspect and any possible implementation thereof; or, when the instructions in the computer-readable storage medium are executed by a processor of a client, enables the client to perform the method provided by the first aspect and any possible implementation thereof.

[0023] The beneficial effects of the second to fourth aspects mentioned above can be referred to the first aspect, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the cluster switching system provided in the embodiments of this application;

[0026] Figure 2 A schematic diagram illustrating the composition of the electronic device provided in the embodiments of this application;

[0027] Figure 3A flowchart illustrating the cluster switching method provided in this application embodiment;

[0028] Figure 4 Another flowchart illustrating the cluster switching method provided in this application embodiment;

[0029] Figure 5 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application;

[0030] Figure 6 A simulation diagram illustrating various consumption scenarios in a Kafka cluster provided in the embodiments of this application;

[0031] Figure 7 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application;

[0032] Figure 8 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the alarm process provided in the embodiments of this application;

[0034] Figure 10 This is a schematic diagram of the cluster switching device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0037] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0038] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0039] As one of the 10 capability centers of the network middleware platform, the fault center system is responsible for processing fault data nationwide and promptly identifying equipment and network problems. It occupies an important position as the data hub of the entire fault platform system. Therefore, in addition to having the function of real-time alarm processing, the fault center system also needs to adopt a protection mechanism to ensure the reliability of core business components.

[0040] Currently, the fault center system uses a distributed messaging system (Kafka cluster) as its core business component.

[0041] Based on this, embodiments of this application provide a cluster switching method, apparatus, device, and storage medium. By analyzing the working status of the primary and backup clusters in the Kafka cluster and calculating the target offset, the execution of the cluster switching operation can be determined, ensuring a smooth business switching process. This solves problems such as untimely data recovery or data loss, and ensures the real-time performance and reliability of the fault center system.

[0042] For ease of understanding, the cluster switching method provided in this application will be described in detail below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic diagram illustrating the composition of a cluster switching system provided in an embodiment of this application. Figure 1 As shown, the cluster switching system includes a cluster switching device 100 and a Kafka cluster 200.

[0044] The cluster switching device 100 can be a server, processor, or other device with switching synchronization service functions. This application embodiment does not limit this, but relevant technologies can be referred to for details, which will not be elaborated here.

[0045] The cluster switching device 100 can be an electronic device with computing and processing capabilities, such as a computer or server.

[0046] The server can be a single server or a server cluster consisting of multiple servers. In some implementations, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, such as a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, and multi-cloud, or any combination thereof.

[0047] The cluster switching device 100 is used to monitor the working status of the Kafka cluster 200 and switch between primary and backup clusters, restart the fault center service, or modify its configuration parameters. The specific process can be referred to the cluster switching method provided in the following embodiments, and will not be repeated here.

[0048] A Kafka cluster 200 consists of multiple distributed messaging systems, specifically including a primary cluster 2001 and a backup cluster 2002.

[0049] Kafka cluster 200 is used to receive configuration data from the Kafka cluster and complete the switchover process. The specific process can be referred to in the following method embodiment, and will not be repeated here.

[0050] The main cluster 2001 is used to process business data, which may include stream processing, real-time analysis, message replay, error recovery, etc. For details, please refer to the relevant technologies, which will not be elaborated here.

[0051] The standby cluster 2002 is used to synchronize business data in the primary cluster 2001. For details, please refer to the relevant technologies, which will not be elaborated here.

[0052] The backup cluster 2002 is also used to process business data in place of the primary cluster when the primary cluster is unavailable. The specific process can be referred to in the following method embodiment, and will not be repeated here.

[0053] The execution entity of the cluster switching method provided in this application embodiment can be the aforementioned cluster switching device 100. As described above, the cluster switching device 100 can be an electronic device with computing processing capabilities, such as a computer or server. Optionally, the cluster switching device 100 can also be a processor in the aforementioned electronic device; or, the data processing device can also be an application (APP) with cluster switching functionality installed in the aforementioned electronic device; or, the cluster switching device 100 can also be a functional module with cluster switching functionality in the aforementioned electronic device; or, the cluster switching device 100 can also be a software system or service platform deployed in the aforementioned electronic device, etc., and this application embodiment does not limit this.

[0054] For simplicity, the following description will use the cluster switching device 100 as an electronic device as an example.

[0055] Figure 2 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device may include: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0056] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0057] Processor 10 is used to execute instructions stored in memory 20 to implement the cluster switching method provided in the following embodiments of this application. Processor 10 may be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. Processor 10 may also be any other device with processing capabilities, such as a circuit, device, or software module; this application embodiment does not limit this. In one example, processor 10 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 in the example. As an optional implementation, the electronic device may include multiple processors; for example, in addition to processor 10, it may also include processor 60. Figure 2 (The example shown is a dashed line).

[0058] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0059] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0060] Communication line 30 is used to transmit information between the components included in the electronic device.

[0061] Communication interface 40 is used to communicate with other devices (such as the Kafka cluster 200 mentioned above) or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0062] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0063] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action interaction or information exchange between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0064] It should be noted that, Figure 2 The structures shown do not constitute a limitation on electronic devices, except... Figure 2 In addition to the components shown, electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different component arrangements.

[0065] The cluster switching method provided in the embodiments of this application will now be described in conjunction with the accompanying drawings.

[0066] Figure 3 This is a flowchart illustrating the cluster switching method provided in an embodiment of this application. Figure 3 As shown, the cluster switching method includes:

[0067] S101: Electronic devices obtain the current working status of the primary cluster and the standby cluster in the Kafka cluster.

[0068] The working status of the primary cluster and the standby cluster includes available and unavailable, respectively.

[0069] Optionally, at the current moment, the working status of the primary cluster and the standby cluster in the Kafka cluster can be either both the primary cluster and the standby cluster are available, or the primary cluster is available and the standby cluster is unavailable, or the primary cluster is unavailable and the standby cluster is available, or both the primary cluster and the standby cluster are unavailable.

[0070] In one possible implementation, the primary and backup clusters in the Kafka cluster can send working status information to the electronic device at a preset period. This working status information can be used to indicate the working status of the primary and backup clusters. The electronic device can determine the current working status of the primary and backup clusters based on the working status information.

[0071] In another possible implementation, the primary and standby clusters in the Kafka cluster can send heartbeat messages (or heartbeat packets) to electronic devices at preset intervals. In this case, the electronic devices can determine the current working status of the primary and standby clusters based on whether they receive the heartbeat message.

[0072] S102. When the working state of the primary cluster is unavailable and the working state of the standby cluster is available, the electronic device acquires the first offset, the second offset, and the third offset.

[0073] The first offset is the maximum offset of the target consumer group in the primary cluster; the second offset is the offset currently consumed by the target consumer group in the primary cluster; and the third offset is the maximum offset of the target consumer group in the standby cluster. The target consumer group can be any one of multiple consumer groups, and the offset currently consumed by the target consumer group is the offset of the consumed data that the target consumer group has already consumed.

[0074] Optionally, prior to S102 above, the electronic device may also obtain the identification (ID) of the target consumer group, and determine the target consumption situation corresponding to the target consumer group based on the correspondence between the identification and consumption situation, and obtain the first offset, the second offset, and the third offset from the target consumption situation.

[0075] The consumption data includes the maximum offset of the corresponding consumer group in the primary cluster, the current consumption offset of the corresponding consumer group in the primary cluster, and the maximum offset of the corresponding consumer group in the standby cluster.

[0076] Optionally, the electronic device can use the identity identifier of the target consumer group as an index to traverse the correspondence between identity identifiers and consumption status, and determine the consumption status corresponding to the identity identifier of the target consumer group as the target consumption status.

[0077] For example, the correspondence between identity identifiers and consumption details can be shown in Table 1 below:

[0078] Table 1

[0079] Identity identifier Consumption situation Identity Token 1 Consumption Situation 1 Identity Token 2 Consumption Situation 2 Identity Token 3 Consumption Situation 3

[0080] As shown in Table 1, this table may include an identity identifier item and a consumption status item. The identity identifier item includes identity identifier 1, identity identifier 2, and identity identifier 3; the consumption status item includes consumption status 1, consumption status 2, and consumption status 3. The consumption status corresponding to identity identifier 1 is consumption status 1; the consumption status corresponding to identity identifier 2 is consumption status 2; and the consumption status corresponding to identity identifier 3 is consumption status 3.

[0081] For example, assuming the identity identifier corresponding to the target consumer group is identity identifier 1, the electronic device can determine that the consumption situation corresponding to identity identifier 1 is consumption situation 1. Taking consumption situation 1 at time t1 (the first offset at time t1 (the maximum offset of the target consumer group in the main cluster) is 100, the second offset at time t1 (the current consumption offset of the target consumer group in the main cluster) is 100, and the third offset at time t1 (the maximum offset of the target consumer group in the standby cluster) is 0) as an example, after producing 50 messages, at time t2, the electronic device can determine that the first offset at time t2 (the maximum offset of the target consumer group in the main cluster) is 100+50=150, and the second offset at time t2 (the current consumption offset of the target consumer group in the main cluster) is 100+50=150; because the standby cluster synchronized the 50 produced messages, the electronic device can determine that the third offset at time t2 (the maximum offset of the target consumer group in the standby cluster) is 0+50=50.

[0082] S103. The electronic device determines the target offset based on the first offset, the second offset, and the third offset.

[0083] The specific process of S103 can be described as follows in S1031 to S1032, and will not be repeated here.

[0084] S104. The electronic device sends the target offset to the backup cluster so that the backup cluster processes the consumed data according to the target offset.

[0085] Optionally, electronic devices can send the target offset to the standby cluster via the fault center service, so that the standby cluster can start processing the consumed data from the target offset + 1.

[0086] For example, taking a target offset of 50 as an example, the electronic device can store the calculated target offset of 50 through the fault center service. When performing the switchover operation, the electronic device sends the target offset of 50 stored in the fault center service to the standby cluster so that the standby cluster can start processing the consumption data from 50+1=51.

[0087] The cluster switching method provided in this application, by analyzing the working status of the primary cluster and the standby cluster in the Kafka cluster and calculating the target offset, can ensure the smoothness of business switching, thereby solving problems such as untimely data recovery or data loss, and ensuring the real-time performance and reliability of the fault center system.

[0088] The following is a description of S103.

[0089] In some possible embodiments, Figure 4 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application. Figure 4 As shown, S103 may specifically include S1031 to S1032.

[0090] S1031. The electronic device determines the fourth offset of the target consumer group in the main cluster that is not currently consumed based on the first offset and the second offset.

[0091] The fourth offset is the offset of the consumption data that the target consumer group in the main cluster has not yet consumed.

[0092] Alternatively, the electronic device may use the difference between the first offset and the second offset as the fourth offset.

[0093] For example, taking time t2 in S102 above as an example, the first offset at time t2 is 150 and the second offset at time t2 is 150. The electronic device can use the difference between the first offset and the second offset, i.e., 150-150=0, as the fourth offset of the target consumer group in the main cluster that is not currently consumed, so it can be determined that the fourth offset at time t2 is 0.

[0094] S1032. The electronic device determines the target offset based on the third offset and the fourth offset.

[0095] In some possible embodiments, Figure 5 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application. Figure 5 As shown, S1032 may include S10321.

[0096] S10321, the electronic device uses the difference between the third offset and the fourth offset as the target offset. For example, taking time t2 in S1031 above as an example, the third offset at time t2 is 50 and the fourth offset at time t2 is 0. The electronic device can determine the target offset as the difference between the third offset and the fourth offset, that is, 50-0=50.

[0097] Exemplary, based on the understanding of the above embodiments, Figure 6 This is a simulation diagram illustrating various consumption scenarios in a Kafka cluster provided in an embodiment of this application. For example... Figure 6 As shown, the various consumption scenarios in the Kafka cluster can include the consumption scenarios from time t1 to t6. Here, kafka1 represents the primary cluster, kafka2 represents the backup cluster, endOffset in kafka1 represents the first offset, curOffset in kafka1 represents the second offset, endOffset in kafka2 represents the third offset, and lag represents the fourth offset.

[0098] The initial state of the primary and backup clusters at time t1 is as follows: the first offset at time t1 is 100, the second offset at time t1 is 100, the third offset at time t1 is 0, and the fourth offset at time t1 is equal to the first offset at time t1 minus the second offset at time t1, which is 100 - 100 = 0.

[0099] After producing 50 messages, at time t2, the first offset is 100 + 50 = 150, the second offset is 100 + 50 = 150, the third offset is 0 + 50 (because Kafka2 synchronized the 50 produced messages), and the fourth offset is equal to the first offset - the second offset = 150 - 150 = 0.

[0100] At time t3, Kafka1 fails. The electronic devices obtain the working status of the primary cluster as unavailable and the working status of the standby cluster as available. The target offset is calculated to execute the switchover. The third offset at time t3 is 50, the fourth offset at time t3 is 0, and the target offset at time t3 is ( Figure 6 The curOffset in kafka2 at time t3 is equal to the third offset at time t3, which is equal to the fourth offset at time t3. The total offset is 50 - 0 = 50.

[0101] After another 30 messages are produced, at time t4, the first offset is 50 + 30 = 80. Because there may be a problem of reduced processing efficiency in the electronic device, only 10 of the newly produced 30 messages are processed. Therefore, the second offset at time t4 is 50 + 10 = 60. At this time, Kafka2 is equivalent to the main cluster. The third offset at time t4 is the curOffset in Kafka1, which is 150. The fourth offset at time t4 is the first offset at time t4 - the second offset at time t4 = 80 - 60 = 20.

[0102] After producing 50 more messages, at time t5, Kafka2 fails. Kafka1 then recovers and synchronizes the newly produced 50 messages. Time t5 includes both the period before and after the switchover. Before the switchover, the first offset at time t5 is 80 + 50 = 130. Because there might be reduced processing efficiency in the electronic device, only 30 of the newly produced messages are processed, so the second offset at time t5 is 60 + 30 = 90. Since Kafka1 acts as a backup cluster and synchronizes the newly produced 50 messages, the third offset at time t5 is 150 + 50 = 200. The fourth offset at time t5 = the first offset at time t5 - the second offset at time t5 = 130 - 90 = 40. Because Kafka2 fails, the electronic device obtains the status of the primary cluster as unavailable and the backup cluster as available, calculates the target offset, and performs the switchover. The target offset at time t5 ( Figure 6 In kafka1 at time t5, the first offset is 200 - the second offset is 160, the third offset is 130, and the fourth offset is equal to the first offset - the second offset = 200 - 160 = 40. After the switchover, kafka1 resumes its primary cluster function. Therefore, the first offset at time t5 is 200, the second offset is 160, the third offset is 130, and the fourth offset is equal to the first offset - the second offset = 200 - 160 = 40.

[0103] After producing another 20 messages, at time t6, Kafka2 recovers and synchronizes the newly produced 20 messages. The first offset at time t6 is 200 + 20 = 220. Because there may be a decrease in processing efficiency in the electronic device's processing program, only 10 of the newly produced 20 messages are processed, so the second offset at time t6 is 160 + 10 = 170. Since Kafka2 has recovered and synchronized the newly produced 20 messages, the third offset at time t6 is 130 + 20 = 150. The fourth offset at time t6 = the first offset at time t6 - the second offset at time t6 = 220 - 170 = 50.

[0104] In other possible embodiments, there is a certain time delay in the electronic device acquiring the first, second, and third offsets, which may lead to data duplication and loss. This problem can be solved by a static compensation mechanism. In this case, Figure 7 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application. Figure 7 As shown, S1032 may further include S10322 to S10324.

[0105] S10322. Electronic devices obtain data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment.

[0106] For example, the data traffic in the current network environment can be 1000 messages / second, the monitoring cycle can be 3 seconds / time, and the number of partitions in the Kafka cluster can be 12.

[0107] S10323. Determine the compensation value based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment.

[0108] Alternatively, the electronic device may use the product of the data flow and the first value as a compensation value.

[0109] The first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster.

[0110] For example, taking the current network environment as an example, with a data traffic of 1000 messages / second, a monitoring period of 3 seconds / time, and a Kafka cluster partition number of 12, the electronic device can determine the first value as the ratio of the monitoring period to the Kafka cluster partition number, i.e., 3 / 12 = 0.25; and then determine the compensation value as the product of the data traffic and the first value, i.e., 1000 × 0.25 = 250.

[0111] S10324. The sum of the first target offset and the compensation value is taken as the target offset.

[0112] The first target offset is the difference between the third and fourth offsets.

[0113] For example, taking time t2 in S1031 above as an example, the third offset at time t2 is 50 and the fourth offset at time t2 is 0. Then the electronic device can determine that the first target offset at time t2 is the difference between the third offset at time t2 and the fourth offset at time t2, that is, 50-0=50. At this time, the target offset is equal to the sum of the first target offset at time t2 and the compensation value. Taking the network environment in S2021 as an example, the electronic device can determine that the target offset is 50+250=300.

[0114] In some possible embodiments, as described above, the working states of the primary cluster and the standby cluster include available and unavailable, respectively. In this case, Figure 8 This is another flowchart illustrating the cluster switching method provided in an embodiment of this application. Figure 8 As shown, the method also includes:

[0115] S201. An alarm is issued when the electronic device is unavailable in both the primary cluster and the standby cluster, or when the primary cluster is available but the standby cluster is unavailable.

[0116] For example, the electronic device can be connected to the management device of the administrator. When it is found that the working status of both the primary cluster and the standby cluster is unavailable, or the working status of the primary cluster is available and the working status of the standby cluster is unavailable, the electronic device can specifically issue an alarm by sending alarm information to the management device.

[0117] Based on the understanding of the above embodiments, Figure 9 This is a schematic diagram of the alarm process provided in an embodiment of this application. Figure 9 As shown, the alarm process may include:

[0118] a1. Electronic devices obtain the working status of the primary and backup clusters.

[0119] The electronic device obtains the working status of the primary and backup clusters. If the working status of the primary and backup clusters is that both primary and backup clusters are available, then continue to execute a2; if the working status of the primary and backup clusters is that neither primary nor backup clusters are available, then continue to execute a3; if the working status of the primary and backup clusters is that only one primary or backup cluster is available, then continue to execute a4.

[0120] a2. Both primary and backup clusters are available.

[0121] The electronic device obtains the working status of the primary and backup clusters, indicating that both primary and backup clusters are available, and continues to execute a21.

[0122] a21. Determine if the synchronization service is started.

[0123] The electronic device determines whether the synchronization service of the primary and backup cluster is running. If the synchronization service is running, the program ends; if the synchronization service is not running, it continues to execute a22.

[0124] a22. Start the synchronization program.

[0125] For example, electronic devices can modify the configuration of the Kafka cluster and restart the synchronization program through the fault center service to ensure that the synchronization service of the primary and standby clusters is normal.

[0126] a3. Both the primary and backup clusters are unavailable.

[0127] The electronic device learns that both the primary and backup clusters are unavailable, and continues to execute a31.

[0128] a31. The electronic device issued an alarm.

[0129] Alternatively, electronic devices can directly collect alarm data from the enterprise service bus (ESB), process the alarm data through the processing module, and issue alarms.

[0130] a4. Only one of the primary and backup clusters is available.

[0131] The electronic device obtains the working status of the primary and backup clusters, indicating that only one of the primary and backup clusters is available, and continues to execute a5.

[0132] a5. Stop the synchronization process.

[0133] The electronic device stops the synchronization process, thereby interrupting the synchronization service between the primary and backup clusters, and then executes a6.

[0134] a6. Determine if the primary cluster is unavailable.

[0135] The electronic device determines whether the main cluster is unavailable. If the main cluster is unavailable, it executes a7; otherwise, it executes a31.

[0136] a7. Stop alarm service.

[0137] The electronic device stops alarm service and continues to execute a8.

[0138] a8. Modify the Kafka cluster configuration.

[0139] Optionally, the electronic device can modify the Kafka cluster configuration through the fault center service and continue executing a9.

[0140] a9. Reset the target offset of the standby cluster.

[0141] Optionally, the electronic device resets the target offset of the standby cluster through the fault center service. The acquisition of the target offset can be done as described above. Figure 1 As described in S103, it will not be repeated here.

[0142] a10. Restart the alarm service.

[0143] The electronic device restarted the alarm service and continued to execute a11.

[0144] a11. Send notification.

[0145] Alternatively, electronic devices can send notifications to consumers via a shared message queue in the Kafka cluster, informing them that the consumer program has returned to normal.

[0146] In some possible implementations, high latency may occur during synchronization between the primary and backup clusters, potentially leading to duplicate data consumption. This issue can be addressed at the implementation level of the business modules. In this case, the cluster switching method further includes: electronic devices processing duplicate data consumption.

[0147] Optionally, the electronic device can overwrite and update the duplicate consumption data to resolve primary key conflicts, or filter the duplicate consumption data through a Bloom filter. This application embodiment does not limit this, and relevant technologies can be referred to for details, which will not be elaborated here.

[0148] In some possible implementations, high latency may occur during synchronization between the primary and backup clusters, potentially leading to data loss. This issue can be resolved by data replay. In this case, the cluster switching method further includes: electronic devices processing the lost synchronization data.

[0149] Optionally, the electronic device can establish a replay interface to replay the data lost during synchronization to the target cluster being switched. For details, please refer to the relevant technologies, which will not be elaborated here.

[0150] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] In an exemplary embodiment, this application also provides a cluster switching device. Figure 10 This is a schematic diagram illustrating the composition of the cluster switching device provided in an embodiment of this application. Figure 10 As shown, the device includes:

[0152] Acquisition module 1001 and processing module 1002.

[0153] The acquisition module 1001 is used to acquire the working status of the primary cluster and the standby cluster in the Kafka cluster at the current time; when the working status of the primary cluster is unavailable and the working status of the standby cluster is available, it acquires the first offset, the second offset, and the third offset.

[0154] The processing module 1002 is used to determine the target offset based on the first offset, the second offset, and the third offset; and send the target offset to the standby cluster so that the standby cluster processes the consumed data according to the target offset.

[0155] In some possible embodiments, the acquisition module 1001 is specifically used to acquire the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment.

[0156] In some possible embodiments, the processing module 1002 is specifically used to determine the fourth offset of the target consumer group in the main cluster that is not currently consumed based on the first offset and the second offset; and to determine the target offset based on the third offset and the fourth offset.

[0157] In some possible embodiments, the processing module 1002 is specifically configured to use the difference between the third offset and the fourth offset as the target offset.

[0158] In some possible embodiments, the processing module 1002 is further configured to determine the compensation value based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; and to use the sum of the first target offset and the compensation value as the target offset; the first target offset is the difference between the third offset and the fourth offset.

[0159] In some possible embodiments, the processing module 1002 is further configured to use the product of the data flow and a first value as a compensation value; the first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster.

[0160] In some possible embodiments, the processing module 1002 is further configured to issue an alarm when both the primary cluster and the standby cluster are unavailable, or when the primary cluster is available and the standby cluster is unavailable.

[0161] In an exemplary embodiment, this application also provides a computer program product that, when run on a computer, causes the computer to execute the aforementioned related method steps to implement the cluster switching method in the above embodiments.

[0162] In an exemplary embodiment, this application also provides a computer-readable storage medium storing computer program instructions thereon; when the computer program instructions are executed by an electronic device, the electronic device causes the electronic device to perform the method described in the foregoing embodiments. The computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0163] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0164] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0165] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cluster switching method, characterized in that, The method includes: Get the current working status of the primary cluster and the standby cluster in the Kafka cluster, respectively. When the primary cluster is unavailable and the backup cluster is available, a first offset, a second offset, and a third offset are obtained; the first offset is the maximum offset of the target consumer group in the primary cluster; the second offset is the offset currently consumed by the target consumer group in the primary cluster; and the third offset is the maximum offset of the target consumer group in the backup cluster. Based on the first offset and the second offset, determine the fourth offset of the target consumer group in the main cluster that is currently not consumed; The target offset is determined based on the third offset and the fourth offset; The target offset is sent to the backup cluster so that the backup cluster processes the consumed data according to the target offset.

2. The method according to claim 1, characterized in that, Determining the target offset based on the third offset and the fourth offset includes: The difference between the third offset and the fourth offset is taken as the target offset.

3. The method according to claim 1, characterized in that, Determining the target offset based on the third offset and the fourth offset includes: Obtain the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; The compensation value is determined based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment. The sum of the first target offset and the compensation value is taken as the target offset; the first target offset is the difference between the third offset and the fourth offset.

4. The method according to claim 3, characterized in that, The process of determining the compensation value based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment includes: The compensation value is the product of the data traffic and the first value; the first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster.

5. The method according to any one of claims 1-4, characterized in that, The working status of the primary cluster and the working status of the standby cluster include available and unavailable, respectively. The method further includes: An alarm is issued when both the primary cluster and the standby cluster are unavailable, or when the primary cluster is available but the standby cluster is unavailable.

6. A cluster switching device, characterized in that, The device includes: an acquisition module and a processing module; The acquisition module is used to acquire the current working status of the primary cluster and the standby cluster in the Kafka cluster; when the working status of the primary cluster is unavailable and the working status of the standby cluster is available, it acquires a first offset, a second offset, and a third offset; the first offset is the maximum offset of the target consumer group in the primary cluster; the second offset is the offset currently consumed by the target consumer group in the primary cluster; and the third offset is the maximum offset of the target consumer group in the standby cluster. The processing module is configured to determine, based on the first offset and the second offset, a fourth offset in which the target consumer group in the primary cluster is currently not consuming data; determine a target offset based on the third offset and the fourth offset; and send the target offset to the backup cluster so that the backup cluster processes the consumed data based on the target offset.

7. The apparatus according to claim 6, characterized in that, The acquisition module is specifically used to acquire the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment. The processing module is specifically configured to use the difference between the third offset and the fourth offset as the target offset; or, The processing module is specifically configured to perform at least one of the following: The compensation value is determined based on the data traffic, monitoring period, and number of partitions in the Kafka cluster under the current network environment; the sum of the first target offset and the compensation value is taken as the target offset; the first target offset is the difference between the third offset and the fourth offset; The compensation value is the product of the data traffic and the first value; the first value is the ratio of the monitoring period to the number of partitions in the Kafka cluster. An alarm is issued when both the primary cluster and the standby cluster are unavailable, or when the primary cluster is available but the standby cluster is unavailable.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores software instructions that, when executed in an electronic device, cause the electronic device to perform the method as described in any one of claims 1-5.

Images