Kafka cluster and message synchronization method and device thereof, electronic equipment and storage medium

Abstract

The present disclosure provides a Kafka cluster and a message synchronization method and device thereof, an electronic device and a storage medium. The method comprises: obtaining a topic filter list saved in a target topic of a source Kafka cluster; wherein the source Kafka cluster comprises the target topic and at least one source topic, and the topic filter list comprises an identifier of a source topic allowed to be synchronized and / or an identifier of a source topic not allowed to be synchronized; obtaining the identifier of the source topic allowed to be synchronized and / or filtering out the identifier of the source topic not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filter list, to obtain an identifier of a source topic to be synchronized; and synchronizing messages in the source topic corresponding to the identifier of the source topic to be synchronized to a target Kafka cluster, to back up the messages in the source topic to be synchronized to the Kafka cluster. According to the embodiments of the present disclosure, the problem that the synchronization channel is interrupted due to the need to restart the synchronization component when the configuration of the synchronization component is changed can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a Kafka cluster and its message synchronization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the internet, various terminal applications are constantly being updated and iterated. To achieve communication between applications, Kafka, a distributed publish-subscribe messaging system, is often used as a message queue middleware for distributed message passing, enabling asynchronous message delivery between applications. In the era of big data, the reliance on Kafka has increased even further, making it a crucial core component in applications.

[0003] To ensure high availability and usability across multiple data centers, a synchronization component (MirrorMaker) was introduced as a data synchronization tool for Kafka clusters. This tool creates mirrored data between multiple Kafka clusters. MirrorMaker addresses the issue of prolonged business interruptions under extreme conditions, ensuring rapid application recovery and message data reliability. During MirrorMaker deployment, the topics requiring message synchronization and backup are determined based on actual business needs. The MirrorMaker component is then configured to filter the topic list to obtain and synchronize the topics required for backup.

[0004] In actual production applications, the topics that need to be backed up synchronously are constantly changing over time and with changing requirements. Under the current implementation of the MirrorMaker component, manual changes to the MirrorMaker component configuration are required, along with restarting the MirrorMaker component cluster. During this process, the synchronization channel will be interrupted for a certain period of time, which reduces the effectiveness of the application's high availability guarantee. Summary of the Invention

[0005] This disclosure provides a Kafka cluster and its message synchronization method, apparatus, electronic device, and storage medium to solve the problem of synchronization channel interruption caused by restarting the synchronization component when the configuration of the synchronization component is changed.

[0006] Firstly, this disclosure provides a message synchronization method for a Kafka cluster, including:

[0007] Obtain the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes the target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of corresponding categories, and the topic filtering list includes identifiers of source topics that are allowed to be synchronized and / or identifiers of source topics that are not allowed to be synchronized;

[0008] Based on the topic filtering list, obtain the identifiers of the source topics that are allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster and / or filter out the identifiers of the source topics that are not allowed to be synchronized, to obtain the identifiers of the source topics to be synchronized.

[0009] The messages in the source topic corresponding to the identifier of the source topic to be synchronized are synchronized to the target Kafka cluster, so as to back up the messages in the source topic to be synchronized to the Kafka cluster.

[0010] Secondly, this disclosure provides a message synchronization device for a Kafka cluster, including:

[0011] The acquisition module is used to acquire the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes the target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of corresponding categories, and the topic filtering list includes the identifiers of source topics that are allowed to be synchronized and / or the identifiers of source topics that are not allowed to be synchronized;

[0012] The filtering module is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filtering list, so as to obtain the identifiers of the source topics to be synchronized.

[0013] The synchronization module is used to synchronize messages in the topic to be synchronized to the target Kafka cluster.

[0014] Thirdly, this disclosure provides a Kafka cluster, including a target topic and at least one source topic;

[0015] The target topic is used to store a topic filtering list, wherein the topic filtering list includes identifiers of source topics that are allowed to be synchronized and / or identifiers of source topics that are not allowed to be synchronized.

[0016] The source topic is used to store messages of the corresponding category;

[0017] The topic filtering list is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or to filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster, so as to obtain the identifiers of the source topics to be synchronized, and synchronize the messages in the source topics corresponding to the identifiers of the source topics to be synchronized to the target Kafka cluster, so as to back up the messages in the source topics to be synchronized to the Kafka cluster.

[0018] Fourthly, this disclosure provides an electronic device, including:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores one or more computer programs that can be executed by the at least one processor, such that the at least one processor can perform the message synchronization method for the Kafka cluster as described in the first aspect.

[0022] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described message synchronization method for a Kafka cluster.

[0023] As can be seen, the embodiments provided in this disclosure, by setting a target topic in the source Kafka cluster and storing a topic filtering list in the target topic, and configuring the identifiers of source topics that are allowed to be synchronized and / or the identifiers of source topics that are not allowed to be synchronized in the topic filtering list, when synchronizing messages in the source Kafka cluster, the topic filtering list is obtained from the target topic, and the source topics in the source Kafka cluster are filtered according to the topic filtering list to obtain the identifiers of source topics that are allowed to be synchronized and / or filter out the identifiers of the source topics that are not allowed to be synchronized, i.e., the identifiers of the source topics to be synchronized, and the source topics corresponding to the identifiers of the source topics to be synchronized are synchronized to the target Kafka cluster to achieve the purpose of backup. The entire message synchronization process does not require changing the configuration file of the synchronization component. The component to be synchronized can be obtained by reading and filtering the target topic. Since there is no need to modify the configuration file of the synchronization component, there is no need to restart the synchronization component due to configuration file changes, which would cause the synchronization channel to be interrupted. Thus, hot loading of Kafka cluster synchronization is achieved.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0026] Figure 1 This is a diagram illustrating an application scenario for message synchronization in a Kafka cluster, provided by an embodiment of this disclosure.

[0027] Figure 2 This is a schematic diagram of a message synchronization method for a Kafka cluster provided in an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram illustrating the overall process of message synchronization in a Kafka cluster, provided in an embodiment of this disclosure.

[0029] Figure 4 A block diagram of a message synchronization device for a Kafka cluster provided in an embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram of a Kafka cluster provided in an embodiment of the present disclosure;

[0031] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0034] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0036] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0037] In developing this disclosure, the inventors discovered that during production operations, it is unavoidable to disable or add synchronous backups for certain topics. Achieving this requires modifying the MirrorMaker component's configuration file and restarting the MirrorMaker cluster. This process is entirely manual, making it impossible to guarantee the correctness of the configuration changes. The lack of rule-based validation makes it highly prone to errors, potentially leading to configurations not taking effect promptly or causing more serious production incidents, resulting in the MirrorMaker component being unable to function properly for extended periods. Furthermore, after changing the configuration, a manual restart of the MirrorMaker cluster is required to ensure it obtains the latest configuration and performs backups accordingly. Due to the inherent mechanism of the MirrorMaker component, the restart process includes a lengthy initialization phase, causing the synchronization component to be unable to recover for an extended period, thus compromising the high availability of the entire Kafka cluster.

[0038] To address the issue that changes to the MirrorMaker component's configuration file can compromise the high availability of a Kafka cluster, this disclosure provides a message synchronization method for a Kafka cluster to ensure its high availability.

[0039] First, the technical terms involved in the embodiments of this disclosure are explained as follows:

[0040] Kafka is a high-throughput distributed publish-subscribe messaging system capable of processing all action stream data from consumers on a website;

[0041] A Kafka cluster consists of multiple Kafka instances (or nodes) running on one or more servers;

[0042] The MirrorMaker component is a synchronization component used to create mirrored data between Kafka clusters;

[0043] A MirrorMaker cluster consists of multiple MirrorMaker components;

[0044] A topic is a logical container that carries messages. In practice, it is often used to distinguish specific business processes. Topics in a Kafka cluster often adopt a multi-subscriber pattern, where one or more consumers can subscribe to the data in a topic. Messages are the main objects processed by Kafka.

[0045] Hot reloading is a technique that responds to changes without restarting the service.

[0046] Figure 1 The diagram illustrates an application scenario for message synchronization of a Kafka cluster provided in this embodiment. This application scenario includes a source Kafka cluster 10, a target Kafka cluster 11, and a MirrorMaker cluster 12, which includes multiple MirrorMaker components 13.

[0047] The source Kafka cluster 10 runs on one or more servers, and the target Kafka cluster 11 also runs on one or more servers. The MirrorMaker cluster 12 connects the source Kafka cluster 10 and the target Kafka cluster 11, and synchronizes messages from the source Kafka cluster 10 to the target Kafka cluster 11 in real time.

[0048] The MirrorMaker cluster 12 can be connected to the source Kafka cluster 10 and the target Kafka cluster 11 via a network, which can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0049] The servers running on the source Kafka cluster 10 and the target Kafka cluster 11 can be servers that provide various services, such as a backend management server that supports websites browsed by users using their terminal devices (this is just an example). The backend management server can analyze and process data such as received user requests.

[0050] It should be understood that Figure 1 The number of source Kafka clusters 10, target Kafka clusters 11, and MirrorMaker clusters 12 shown in the diagram is merely illustrative. Depending on implementation requirements, there can be any number of source Kafka clusters 10, target Kafka clusters 11, and MirrorMaker clusters 12.

[0051] Figure 2 This is a schematic flowchart illustrating a message synchronization method for a Kafka cluster provided in an embodiment of this disclosure. The method is executed by a synchronization component, which includes, but is not limited to, the MirrorMaker component. (Refer to...) Figure 2 As shown, the method mainly includes:

[0052] Step 201: Obtain the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes a target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of the corresponding categories, and the topic filtering list includes the identifiers of the source topics that are allowed to be synchronized and / or the identifiers of the source topics that are not allowed to be synchronized.

[0053] The topic filter list is configured with topic filtering rules, which identify the source topics that are allowed to be synchronized and / or those that are not. If no topic filter list is configured or no filtering rules are required, the topic filter list can be empty, meaning it does not include any source topic identifiers.

[0054] In some embodiments, the topic filtering list includes a whitelist filtering list and / or a blacklist filtering list; obtaining the topic filtering list stored in the target topic of the source Kafka cluster includes: obtaining the whitelist filtering list stored in the first target topic of the Kafka cluster, the whitelist filtering list including identifiers of source topics that are allowed to be synchronized; and / or, obtaining the blacklist filtering list stored in the second target topic of the Kafka cluster, the blacklist filtering list including identifiers of source topics that are not allowed to be synchronized.

[0055] In some embodiments, the number of target topics used to store the topic filtering list can be one or two. When the first target topic and the second target topic are the same topic, a mapping relationship between the source topic identifier and the filtering tag is set in the topic filtering list. The filtering tag includes a whitelist tag and a blacklist tag. When the filtering tag corresponding to the source topic identifier is a whitelist tag, it means that the source topic corresponding to the source topic identifier needs to be synchronized. When the filtering tag corresponding to the source topic identifier is a blacklist tag, it means that the source topic corresponding to the source topic identifier does not need to be synchronized. Topics with whitelist tags in the filtering list constitute the whitelist filtering list, and topics with blacklist tags constitute the blacklist filtering list. When the first target topic and the second target topic are different topics, one target topic is used to store the whitelist filtering list, and the other target topic is used to store the blacklist filtering list.

[0056] By configuring a theme filter list (i.e., filter rules) in the target theme, instead of setting filter rules in the synchronization component's configuration file, this approach allows for easy updates to the filter rules. Only the filter list in the target theme needs modification, eliminating the need to change the synchronization component's configuration file and thus avoiding the need to restart the synchronization component due to configuration changes. Furthermore, configuring the theme filter list in the target theme does not require modifications to other components, avoiding external dependencies during the synchronization backup process, reducing risks, and ensuring stability.

[0057] In some embodiments, obtaining the topic filter list stored in the target topic of the source Kafka cluster includes: periodically obtaining the topic filter list stored in the target topic of the source Kafka cluster. Every time a timer sets a specified interval, the topic filter list stored in the target topic of the source Kafka cluster is obtained. The timer interval can be configured according to the scenario requirements, for example, the timer can be set to one month, six months, one year, etc., specifically determined by the frequency of synchronization backups. The duration of this timer can also be periodic, that is, the topic filter list stored in the target topic of the source Kafka cluster is obtained every set period. For example, the timer can be set to a period of one month, six months, or one year.

[0058] In an exemplary embodiment, to facilitate the configuration and updating of the topic filtering list, an interactive interface for configuring the topic filtering list is provided. The current configuration of the topic filtering list is displayed through the interactive interface, and the configuration of the topic filtering list is updated based on the displayed content. For example, topics can be added or deleted from the whitelist filtering list of the topic filtering list, and topics can be added or deleted from the blacklist of the blacklist filtering list of the topic filtering list.

[0059] Step 202: Based on the topic filtering list, obtain the identifiers of the source topics that are allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster and / or filter out the identifiers of the source topics that are not allowed to be synchronized, to obtain the identifiers of the source topics to be synchronized.

[0060] In some embodiments, the method for obtaining the topics to be synchronized varies depending on the topic filtering list. The main methods are described in the following three cases:

[0061] Case 1: The topic filter list includes a whitelist filter list.

[0062] In an exemplary embodiment, obtaining the identifier of the allowed source topic from the identifiers of source topics in the source Kafka cluster, based on the topic filtering list, and obtaining the identifier of the source topic to be synchronized includes: when the whitelist filtering list is empty, determining the identifier of the source topic in the source Kafka cluster as the topic to be synchronized; when the whitelist filtering list is not empty, obtaining the identifier of the source topic belonging to the whitelist filtering list from the identifiers of source topics in the source Kafka cluster, and using it as the identifier of the source topic to be synchronized.

[0063] Specifically, when the whitelist filter list is empty, the identifiers of all source topics in the source Kafka cluster are identified as the topics to be synchronized.

[0064] Scenario 2: The topic filter list includes a blacklist filter list.

[0065] In an exemplary embodiment, filtering out the identifiers of source topics that are not allowed to be synchronized from the identifiers of source topics in the source Kafka cluster according to the topic filtering list to obtain the identifiers of source topics to be synchronized includes: obtaining the identifiers of source topics that do not belong to the blacklist filtering list from the identifiers of source topics in the source Kafka cluster, and using them as the identifiers of source topics to be synchronized.

[0066] Scenario 3: The topic filter list includes a whitelist filter list and a blacklist filter list.

[0067] In an exemplary embodiment, obtaining the identifiers of the source topics that are allowed to be synchronized and / or filtering out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filtering list, to obtain the identifiers of the source topics to be synchronized, includes: obtaining the identifiers of the source topics that belong to the whitelist filtering list from the identifiers of the source topics in the source Kafka cluster to obtain a set of identifiers of the source topics that are allowed to be synchronized; and obtaining the identifiers of the source topics that do not belong to the blacklist filtering list from the set of identifiers of the source topics that are allowed to be synchronized, as the identifiers of the source topics to be synchronized.

[0068] Step 203: Synchronize the messages in the source topic corresponding to the identifier of the source topic to be synchronized to the target Kafka cluster, so as to back up the messages in the source topic to be synchronized to the Kafka cluster.

[0069] Figure 3 The diagram illustrates the overall message synchronization process of a Kafka cluster according to an embodiment of this disclosure. The source Kafka cluster includes the following source topics: Topic1, Topic2, ..., Topicn. In addition, two target topics are set in the source Kafka cluster: mm2-white-list.internal for storing the whitelist filter list and mm2-black-list.internal for storing the blacklist filter list. A scheduled task is added to the MirrorMaker component, along with two steps for reading and processing the blacklist and whitelist filter lists. The scheduled task periodically triggers the steps of reading the blacklist and whitelist filter lists. After obtaining the whitelist and blacklist filter lists, the source topics are filtered sequentially using the whitelist and blacklist filter lists to obtain the identifier of the final source topic to be synchronized.

[0070] When synchronization backup requirements change, the whitelist and blacklist filter lists are updated. This includes adding or deleting identifiers for source topics in the whitelist filter list, and similarly, adding or deleting identifiers for source topics in the blacklist filter list. Updating the whitelist and blacklist filter lists can be done using other components or terminals; for example, technicians can update the whitelist and blacklist filter lists using the user interface provided by the terminal. This ensures that the new synchronization backup configuration takes effect promptly without requiring a restart of the MirrorMaker component after a change in synchronization backup requirements.

[0071] Reference Figure 3 As shown, the process of MirrorMaker component performing synchronous backup mainly includes:

[0072] Step 301: The MirrorMaker component obtains the complete list of topics in the source Kafka cluster. This complete list of topics includes the identifiers of all source topics in the source Kafka cluster.

[0073] Step 302: After the scheduled task reaches its scheduled time, the MirrorMaker component reads the whitelist filter list from the mm2-white-list.internal topic in the source Kafka cluster. It then filters the source topic identifiers in the full list of topics according to the identifiers in the whitelist filter list to obtain the identifiers of the source topics allowed for synchronization. Specifically, if the whitelist filter list is not empty, the identifiers of the source topics that exist in both the whitelist filter list and the full list of topics are used as the identifiers of the source topics allowed for synchronization. If the whitelist filter list is empty, the identifiers of the source topics in the full list of topics are used as the identifiers of the source topics allowed for synchronization.

[0074] Step 303: After the scheduled task arrives at the scheduled time, the MirrorMaker component reads the blacklist filter list in the mm2-black-list.internal topic of the source Kafka cluster, and filters the source topic identifiers that are allowed to be synchronized obtained in step 302 according to the source topic identifiers in the blacklist filter list. The blacklist filter list identifiers are removed from the source topic identifiers that are allowed to be synchronized, and the source topic identifiers that are allowed to be synchronized are obtained after being filtered again.

[0075] In step 304, the MirrorMaker component will synchronize messages from each topic corresponding to the identifier of the source topic to the target Kafka cluster.

[0076] In steps 302 and 303, if the scheduled task has not yet arrived, it is not necessary to execute the steps of reading the whitelist filter list from the mm2-white-list.internal topic in the source Kafka cluster, nor is it necessary to execute the steps of reading the blacklist filter list from the mm2-black-list.internal topic in the source Kafka cluster. Only the filtering process based on the current whitelist and blacklist filter lists needs to be executed. The execution order of steps 302 and 303 is not limited; it can be either filtering using the whitelist filter list first, followed by the blacklist filter list, or vice versa. The specific logic can be set according to actual needs.

[0077] In a specific example, suppose that when a new Kafka cluster is connected to an application and a synchronization task is created for the MirrorMaker component, the source Kafka cluster stores identifiers for 10 source topics, from Topic 1 to Topic 10. In the initial scenario, the whitelist and blacklist filter lists are empty. In this scenario, the MirrorMaker component synchronizes all source topics from the source Kafka cluster to the target Kafka cluster. Suppose the requirement changes to no longer synchronizing messages from Topic 1 to Topic 5. Based on this requirement, the identifiers of Topic 1 to Topic 5 are added to the blacklist filter list. A scheduled task within the MirrorMaker component triggers the reading and updating of the whitelist and blacklist filter lists. The whitelist filter list is empty, meaning that Topics 1 to Topic 10 all need to be synchronized. The blacklist filter list contains the identifiers of Topics 1 to Topic 5. Therefore, Topics 1 to Topic 5 are filtered out from the Topics 1 to Topic 10 that need to be synchronized. Finally, the MirrorMaker component synchronizes Topics 6 to Topic 10 to the target Kafka cluster.

[0078] If, after a period of operation, the requirement changes to synchronizing Topic 3 to Topic 8, that is, adding the identifiers of Topic 3 to Topic 8 to the whitelist filter list, both the whitelist and blacklist filter lists will now contain the identifiers of Topic 3 to Topic 5. In this scenario, after the scheduled task within the MirrorMaker component is triggered, it first reads the whitelist filter list and filters it to obtain the identifiers of Topic 3 to Topic 8 as the source topics to be synchronized. Then, it reads the blacklist filter list and filters it to obtain the identifiers of Topic 6 to Topic 8 as the final topics to be synchronized. The MirrorMaker component then synchronizes Topic 6 to Topic 8 from the source Kafka cluster to the target Kafka cluster.

[0079] By configuring whitelist and blacklist filter lists, and through scheduled tasks within the MirrorMaker component, hot reloading of synchronization requirement changes is achieved, avoiding cluster restarts after synchronization requirement changes.

[0080] The embodiments provided in this disclosure set a target topic in the source Kafka cluster and save a topic filtering list in the target topic. The topic filtering list is configured with identifiers of source topics that are allowed to be synchronized and / or identifiers of source topics that are not allowed to be synchronized. When synchronizing messages in the source Kafka cluster, the topic filtering list is obtained from the target topic, and the source topics in the source Kafka cluster are filtered according to the topic filtering list. This obtains the identifiers of source topics that are allowed to be synchronized and / or filters out the identifiers of source topics that are not allowed to be synchronized, i.e., the identifiers of the source topics to be synchronized. The source topics corresponding to the identifiers of the source topics to be synchronized are then synchronized to the target Kafka cluster to achieve the purpose of backup. The entire message synchronization process does not require changes to the configuration file of the synchronization component. The component to be synchronized can be obtained by reading and filtering the target topic. Since there is no need to modify the configuration file of the synchronization component, there is no need to restart the synchronization component due to changes in the configuration file, which would cause the synchronization channel to be interrupted. This achieves hot reloading of Kafka cluster synchronization.

[0081] In summary, this disclosure improves the configuration changes of synchronization components such as MirrorMaker during operation and maintenance. By adding a built-in Topic to the Kafka cluster and periodically loading this built-in Topic in the synchronization component, hot reloading after configuration changes is achieved. This allows for relatively real-time response to business change requirements without restarting the cluster, avoiding frequent restarts during configuration changes and achieving zero restarts for the synchronization component. This effectively ensures the reliability and high availability of the Kafka cluster and the entire service. This disclosure replaces the traditional method of manually modifying configurations and restarting, avoiding manual changes to configuration files, reducing the error rate in configuration adjustments, increasing the reliability and accuracy of the configuration change process, reducing uncontrollable risks during changes, and improving operational efficiency.

[0082] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0083] In addition, this disclosure also provides a Kafka cluster and a message synchronization device, electronic device, and computer-readable storage medium for the Kafka cluster. All of the above can be used to implement any of the message synchronization methods for the Kafka cluster provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0084] Figure 4This is a block diagram of a message synchronization device for a Kafka cluster provided in an embodiment of the present disclosure, applied to a synchronization component. Specific implementation of this message synchronization device for the Kafka cluster can be found in the relevant descriptions of the synchronization component in the above embodiments, and will not be repeated here. The device includes:

[0085] The acquisition module 401 is used to acquire the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes the target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of corresponding categories, and the topic filtering list includes the identifiers of source topics that are allowed to be synchronized and / or the identifiers of source topics that are not allowed to be synchronized.

[0086] The filtering module 402 is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filtering list, so as to obtain the identifiers of the source topics to be synchronized.

[0087] Synchronization module 403 is used to synchronize messages in the topic to be synchronized to the target Kafka cluster.

[0088] Figure 5 This diagram illustrates a Kafka cluster according to an embodiment of this disclosure. For a detailed implementation of this Kafka cluster, please refer to the description of the source Kafka cluster in the above embodiments; it will not be repeated here. The Kafka cluster includes a target topic 502 and a source topic 501. The target topic 502 is used to store a topic filtering list. For details regarding the topic filtering list, please refer to the description in the above embodiments; it will not be repeated here.

[0089] The source topic is used to store messages of the corresponding category;

[0090] The topic filtering list is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or to filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster, so as to obtain the identifiers of the source topics to be synchronized, and synchronize the messages in the source topics corresponding to the identifiers of the source topics to be synchronized to the target Kafka cluster, so as to back up the messages in the source topics to be synchronized to the Kafka cluster.

[0091] Figure 6This is a block diagram of an electronic device provided in an embodiment of the present disclosure. The electronic device includes: at least one processor 601; at least one memory 602; and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs executable by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 to enable the at least one processor 601 to execute the aforementioned message synchronization method for a Kafka cluster.

[0092] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned message synchronization method for a Kafka cluster. The computer-readable storage medium can be volatile or non-volatile.

[0093] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described message synchronization method for the Kafka cluster.

[0094] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0095] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0096] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0097] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0098] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0099] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0103] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A message synchronization method for a Kafka cluster, characterized in that, include: Obtain the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes the target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of corresponding categories, and the topic filtering list includes identifiers of source topics that are allowed to be synchronized and / or identifiers of source topics that are not allowed to be synchronized; Based on the topic filtering list, obtain the identifiers of the source topics that are allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster and / or filter out the identifiers of the source topics that are not allowed to be synchronized, to obtain the identifiers of the source topics to be synchronized. The messages in the source topic corresponding to the identifier of the source topic to be synchronized are synchronized to the target Kafka cluster, so as to back up the messages in the source topic to be synchronized to the Kafka cluster.

2. The message synchronization method for a Kafka cluster according to claim 1, characterized in that, The topic filtering list includes a whitelist filtering list and / or a blacklist filtering list; The process of obtaining the topic filtering list stored in the target topic of the source Kafka cluster includes: Obtain the whitelist filter list stored in the first target topic of the Kafka cluster, the whitelist filter list including the identifiers of the source topics that are allowed to be synchronized; And / or, Obtain the blacklist filter list stored in the second target topic of the Kafka cluster. The blacklist filter list includes the identifiers of source topics that are not allowed to be synchronized.

3. The message synchronization method for a Kafka cluster according to claim 2, characterized in that, The topic filtering list includes a whitelist filtering list; Based on the topic filtering list, the identifiers of the source topics allowed for synchronization are obtained from the identifiers of the source topics in the source Kafka cluster, and the identifiers of the source topics to be synchronized are obtained, including: When the whitelist filter list is empty, the identifier of the source topic in the source Kafka cluster is determined as the identifier of the source topic to be synchronized; When the whitelist filter list is not empty, the identifier of the source topic belonging to the whitelist filter list is obtained from the identifier of the source topic in the source Kafka cluster, and the identifier of the source topic to be synchronized is determined.

4. The message synchronization method for a Kafka cluster according to claim 2, characterized in that, The topic filtering list includes a blacklist filtering list; Based on the topic filtering list, the identifiers of source topics that are not allowed to be synchronized are filtered out from the identifiers of source topics in the source Kafka cluster to obtain the identifiers of source topics to be synchronized, including: From the identifiers of the source topics in the source Kafka cluster, obtain the identifiers of the source topics that do not belong to the blacklist filter list, and determine them as the identifiers of the source topics to be synchronized.

5. The message synchronization method for a Kafka cluster according to claim 2, characterized in that, The topic filtering list includes a whitelist filtering list and a blacklist filtering list; The step of obtaining the identifiers of the source topics to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filtering list includes: From the identifiers of the source topics in the source Kafka cluster, obtain the identifiers of the source topics belonging to the whitelist filter list, and obtain the set of identifiers of the source topics that are allowed to be synchronized; From the set of identifiers of the source topics that are allowed to be synchronized, obtain the identifiers of the source topics that do not belong to the blacklist filter list, and use them as the identifiers of the source topics to be synchronized.

6. The message synchronization method for a Kafka cluster according to any one of claims 1-5, characterized in that, The process of obtaining the topic filtering list stored in the target topic of the source Kafka cluster includes: Periodically retrieve the topic filter list stored in the target topic of the source Kafka cluster.

7. A message synchronization device for a Kafka cluster, characterized in that, include: The acquisition module is used to acquire the topic filtering list stored in the target topic of the source Kafka cluster; wherein, the source Kafka cluster includes the target topic and at least one source topic, the target topic is used to store the topic filtering list, the source topic is used to store messages of corresponding categories, and the topic filtering list includes the identifiers of source topics that are allowed to be synchronized and / or the identifiers of source topics that are not allowed to be synchronized; The filtering module is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster according to the topic filtering list, so as to obtain the identifiers of the source topics to be synchronized. The synchronization module is used to synchronize messages in the topic to be synchronized to the target Kafka cluster.

8. A Kafka cluster, characterized in that, Includes the target topic and at least one source topic; The target topic is used to store a topic filtering list, wherein the topic filtering list includes identifiers of source topics that are allowed to be synchronized and / or identifiers of source topics that are not allowed to be synchronized. The source topic is used to store messages of the corresponding category; The topic filtering list is used to obtain the identifiers of the source topics that are allowed to be synchronized and / or to filter out the identifiers of the source topics that are not allowed to be synchronized from the identifiers of the source topics in the source Kafka cluster, so as to obtain the identifiers of the source topics to be synchronized, and synchronize the messages in the source topics corresponding to the identifiers of the source topics to be synchronized to the target Kafka cluster, so as to back up the messages in the source topics to be synchronized to the Kafka cluster.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, such that the at least one processor can perform the message synchronization method for the Kafka cluster as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the message synchronization method for the Kafka cluster as described in any one of claims 1-6.

Images