Real-time regulation and control method and device for number of Kafka cluster copies
By monitoring the replica distribution and business performance indicators of the Kafka cluster in real time and automatically adjusting the replica storage, the system instability problem caused by abnormal replica counts of Kafka cluster is solved, the recovery efficiency and system stability are improved, and the operation and maintenance costs are reduced.
Patent Information
- Application Number
- CN202410914442.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-12
AI Technical Summary
The number of Kafka cluster replicas changes due to node failures and network partitions, resulting in a decrease in system reliability and stability. The existing technology relies on manual monitoring and adjustment, which is inefficient and has high maintenance costs.
By obtaining the replica distribution of each partition under the topic in real time, automatically identifying replica exceptions, dynamically adjusting replica storage according to preset thresholds and business performance indicators, and selecting Brokers with lower loads to reduce the impact of manual intervention and system instability.
It improves the efficiency of replica count recovery, reduces operation and maintenance costs, enhances the degree of automation and stability of the system, and reduces the impact of fluctuations in business performance indicators on user experience.
Smart Images

Figure CN118939184A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Kafka cluster replica management, and in particular to a method and device for real-time control of the number of Kafka cluster replicas. Background Art
[0002] Kafka cluster is a distributed stream processing platform, which is widely used in various real-time message processing scenarios such as log aggregation, indicator monitoring, user activity tracking, real-time analysis, stream processing, etc., with high availability and data reliability. The availability and data reliability of Kafka cluster mainly depend on its replication mechanism, which ensures data security and consistency by maintaining multiple data copies on different containers (Brokers).
[0003] However, in actual use, the number of copies of the Kafka cluster may change due to node failures and network partitions, resulting in a decrease in the number of copies under a partition in the Kafka cluster, leading to a decrease in system reliability and stability. In the face of the above problems, the prior art often uses monitoring tools by operation and maintenance personnel to manually monitor the number of copies of each partition in the Kafka cluster. When an exception occurs, a Json (JavaScript Object Notation) document containing copy recovery information is manually generated, and the adjustment command is manually executed, so that the Kafka cluster can resume normal working state. This method, which manually generates a Json document, is inefficient, and according to different abnormal states, the code logic for generating the Json document needs to be dynamically adjusted, and the maintenance cost is high.
[0004] Therefore, how to automatically adjust the number of replicas of each abnormal partition in the Kafka cluster and ensure the stability of the system is a technical problem that needs to be solved at present. Summary of the invention
[0005] The present application provides a method and device for real-time control of the number of replicas of a Kafka cluster, so as to solve the technical problem that when facing a large-scale cluster, manual recovery of the number of replicas of each abnormal partition in the Kafka cluster is inefficient, thereby causing system instability.
[0006] In order to solve the above technical problems, in the first aspect, an embodiment of the present application provides a method for real-time control of the number of replicas of a Kafka cluster, including:
[0007] According to the real-time replica distribution list of each partition under the current topic, obtain all first partitions with abnormal number of replicas under the current topic;
[0008] Determine, according to a first preset threshold and a real-time replica distribution list of each of the first partitions under the current topic, a Broker required to store the replica corresponding to the first partition, so as to obtain a first replica distribution list corresponding to the first partition;
[0009] According to the distribution list of the first copies of the first partitions under the current topic and the business performance indicators of the current system, the copies of the first partitions are dynamically stored in the corresponding Broker in sequence.
[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by automatically and in real time acquiring the copy distribution of each partition under the current theme, accurately identifying the abnormal partitions with insufficient number of copies, and then automatically screening the Brokers required to store the newly added copies by pre-setting the normal number of copies and combining the copy distribution of the current abnormal partition, thereby improving the efficiency of recovering the number of copies of the abnormal partition and reducing the impact of manual intervention on the stability of the system. At the same time, since the system state changes during the process of recovering the number of copies, the system becomes unstable, thereby affecting the business. Therefore, in the process of recovering the number of copies in the abnormal partition according to the determined Broker, by real-time monitoring the business performance indicators of the system, different measures are taken according to different system states, thereby reducing the negative impact on the business.
[0011] In some embodiments of the first aspect of the present application, determining a Broker required to store a replica corresponding to the first partition according to a first preset threshold and a real-time replica distribution list of each first partition under a current topic to obtain a first replica distribution list corresponding to the first partition includes:
[0012] Obtain a Broker set of the Kafka cluster, and obtain all first Brokers storing replicas corresponding to the first partition from the real-time replica distribution list;
[0013] Filter out all the first Brokers from the Broker set, and obtain all the second Brokers that do not store a copy of the first partition;
[0014] According to the first preset threshold, several third Brokers are selected from all the second Brokers, and the first replica distribution list is generated according to all the third Brokers.
[0015] Compared with the prior art, the embodiments of the present application have the following beneficial effects: first, according to the copy distribution in the partition with the current abnormal number of copies, the second Broker that does not store the copy of the abnormal partition is obtained, and then a certain number of third Brokers are selected from all the second Brokers, so that after the final copy is stored in the third Broker, the total number of copies of the abnormal partition, that is, the sum of the first Broker and the third Broker meets the first preset threshold, thereby automatically screening out the first copy distribution list for subsequent storage of copies, thereby improving the degree of automation of the Kafka cluster copy number exception handling process.
[0016] In some embodiments of the first aspect of the present application, the selecting a plurality of third Brokers from all the second Brokers according to the first preset threshold comprises:
[0017] Sort all the second Brokers in ascending order according to the number of stored replicas in the second Broker;
[0018] A plurality of third Brokers are selected in sequence from all the sorted second Brokers until the sum of the number of the first Brokers and the number of the third Brokers is equal to a first preset threshold.
[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by giving priority to Brokers with lower loads, excessive pressure on a single Broker is avoided during the replica replication process, thereby improving the stability of the system during the abnormal handling of the number of replicas in the Kafka cluster.
[0020] In some embodiments of the first aspect of the present application, dynamically storing the replica corresponding to the first partition in the corresponding Broker according to the first replica distribution list of each first partition and the service performance index of the current system includes:
[0021] Obtaining business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.;
[0022] Determining whether the current service performance indicator exceeds a second preset threshold;
[0023] If the business performance indicator exceeds the second preset threshold, the copy storage operation of the current first partition is stopped; otherwise, the copy of the current first partition is stored in each Broker in the first copy distribution list in sequence.
[0024] Compared with the prior art, the embodiments of the present application have the following beneficial effects: since in the process of handling the abnormal number of replicas in the Kafka cluster, the process of copying the replica to the Broker will affect the stability of data transmission in the business system, thereby causing a decline in user experience and further affecting the business performance indicators of the business system. Therefore, when the business performance indicators fluctuate too much during the process of handling the abnormal number of replicas in the Kafka cluster, the stability of the system can be improved by temporarily suspending the replica copy operation and waiting for the business performance indicators to stabilize before continuing the copy operation.
[0025] In some embodiments of the first aspect of the present application, obtaining all first partitions with abnormal number of copies under the current theme according to the real-time copy distribution list of each partition under the current theme includes:
[0026] Traverse the real-time replica distribution list of each partition under the current topic, and obtain the first number of Brokers storing replicas of each partition;
[0027] When the first number corresponding to the partition is less than the first preset threshold, the partition is determined to be the first partition.
[0028] Compared with the prior art, the embodiments of the present application have the following beneficial effects: according to the preset replica threshold, through interface instructions, it is possible to quickly identify in which Brokers the replicas of each partition are stored, thereby determining the number of replicas of each partition and improving the degree of automation of abnormal partition identification.
[0029] In a second aspect, an embodiment of the present application provides a real-time control device for the number of replicas of a Kafka cluster, including: a first partition acquisition module, a first replica distribution list acquisition module, and a replica dynamic storage module;
[0030] The first partition acquisition module is used to acquire all first partitions with abnormal number of copies under the current topic according to the real-time copy distribution list of each partition under the current topic;
[0031] The first replica distribution list acquisition module is used to determine the Broker required to store the replica corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each first partition under the current theme, so as to obtain the first replica distribution list corresponding to the first partition;
[0032] The replica dynamic storage module is used to dynamically store the replicas of each first partition in the corresponding Broker in sequence according to the first replica distribution list of each first partition under the current theme and the business performance indicators of the current system.
[0033] In some embodiments of the second aspect of the present application, the first replica distribution list acquisition module is used to determine the Broker required to store the replica corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each first partition under the current theme, so as to acquire the first replica distribution list corresponding to the first partition, including:
[0034] Obtain a Broker set of the Kafka cluster, and obtain all first Brokers storing replicas corresponding to the first partition from the real-time replica distribution list;
[0035] Filter out all the first Brokers from the Broker set, and obtain all the second Brokers that do not store a copy of the first partition;
[0036] According to the first preset threshold, several third Brokers are selected from all the second Brokers, and the first replica distribution list is generated according to all the third Brokers and all the first Brokers.
[0037] In some embodiments of the second aspect of the present application, the selecting a plurality of third Brokers from all the second Brokers according to the first preset threshold comprises:
[0038] Sort all the second Brokers in ascending order according to the number of stored replicas in the second Broker;
[0039] A plurality of third Brokers are selected in sequence from all the sorted second Brokers until the sum of the number of the first Brokers and the number of the third Brokers is equal to a first preset threshold.
[0040] In some embodiments of the second aspect of the present application, the replica dynamic storage module is used to dynamically store the replicas of each first partition in the corresponding Broker in sequence according to the first replica distribution list of each first partition under the current theme and the business performance index of the current system, including:
[0041] Obtaining business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.;
[0042] Determining whether the current service performance indicator exceeds a second preset threshold;
[0043] If the business performance indicator exceeds the second preset threshold, the copy storage operation of the current first partition is stopped; otherwise, the copy of the current first partition is stored in each Broker in the first copy distribution list in sequence.
[0044] In some embodiments of the second aspect of the present application, the first partition acquisition module is used to acquire all first partitions with abnormal number of copies under the current theme according to the real-time copy distribution list of each partition under the current theme, including:
[0045] Traverse the real-time replica distribution list of each partition under the current topic, and obtain the first number of Brokers storing replicas of each partition;
[0046] When the first number corresponding to the partition is less than the first preset threshold, the partition is determined to be the first partition. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A flowchart of a method for real-time control of the number of replicas of a Kafka cluster provided in some embodiments of the present application;
[0048] Figure 2 A schematic diagram of the structure of a device for real-time control of the number of replicas of a Kafka cluster provided in some embodiments of the present application;
[0049] Figure 3 This is a structural diagram of a real-time control system for the number of replicas of a Kafka cluster provided in some embodiments of the present application. DETAILED DESCRIPTION
[0050] When the number of replicas in the Kafka cluster is abnormal due to node failure or network partition, the operation and maintenance personnel often use monitoring tools to manually monitor the number of replicas in each partition of the Kafka cluster, find the abnormality, and manually generate a Json document containing replica recovery information, and further manually execute adjustment commands to restore the Kafka cluster to normal working state. The above method, which manually generates Json documents, is inefficient and needs to dynamically adjust the code logic for generating Json documents according to different abnormal states, which has high maintenance costs.
[0051] In order to solve the above problems, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0052] Embodiment 1
[0053] Please refer to Figure 1 , a real-time control method for the number of replicas of a Kafka cluster provided in an embodiment of the present application, including S10 to S30, specifically including:
[0054] S10: According to the real-time replica distribution list of each partition under the current topic, all first partitions with abnormal number of replicas under the current topic are obtained.
[0055] In some embodiments of the present application, obtaining all first partitions with abnormal number of copies under the current theme according to the real-time copy distribution list of each partition under the current theme includes:
[0056] Traverse the real-time replica distribution list of each partition under the current topic, and obtain the first number of Brokers storing replicas of each partition;
[0057] When the first number corresponding to the partition is less than the first preset threshold, the partition is determined to be the first partition.
[0058] Preferably, in some embodiments of the present application, the real-time replica distribution list can be obtained by the following preferred method: regularly executing kafka-topics API related commands to obtain detailed information of all partitions under each topic (Topic), including the replica list of each partition and the current ISR (In-Sync Replicas) list. Wherein, each replica of each partition is stored in a different Broker.
[0059] Preferably, in some embodiments of the present application, the specific implementation of determining all first partitions can be performed by the following preferred implementation:
[0060] S11: Compare the ISR list obtained through the kafka-topics API related commands with the configured number of replicas. If the number of ISRs of a partition is less than the configured number of replicas, it means that the number of replicas of the partition is abnormal, and the partition is determined to be the first partition;
[0061] S12: Record detailed information of all first partitions, including the partition ID, the topic it is in, the current number of ISRs, and the expected number of replicas.
[0062] It can be seen from the above embodiments that according to the preset replica threshold, through interface instructions, the replicas of each partition can be quickly identified in which Brokers, thereby determining the number of replicas of each partition, which can improve the automation of abnormal partition identification.
[0063] Preferably, in some embodiments of the present application, the above step S11 can also be performed by the following preferred implementation method:
[0064] S111: Create a trigger through Zabbix, set the trigger name as "Kafka Replica Count Insufficient Alarm", and set the monitoring item of the trigger as the ISR count; the expression of the trigger is: {hostname:kafka.num.last()} < EXPECTED_REPLICA_COUNT, where hostname is the hostname monitoring Kafka, and EXPECTED_NUM is the preset replica count threshold;
[0065] S112: When the expression is true, that is, when the monitored replica count is less than the preset threshold, an alarm is triggered.
[0066] Preferably, in some embodiments of the present application, after S112 triggers an alarm, the alarm information is obtained from the monitoring system at regular intervals, further processed, and the alarm information is sent. Specifically:
[0067] S113: Use the timed task scheduling framework in Java, such as ScheduledExecutorService, to execute the operation of obtaining alarm information at regular intervals. An appropriate time interval can be set according to business requirements, such as obtaining alarm information every 5 minutes;
[0068] S114: In the fetchAlarmInfo method, call the API provided by the monitoring system to obtain alarm information;
[0069] S115: In the handleAlarms method, parse the alarm information obtained from the monitoring system and process the alarms that need to be processed; the alarm information can be in JSON format and needs to be parsed according to the specific format.
[0070] Furthermore, the alarm information should at least include: alarm level (such as severe, warning, information); the name of the affected Kafka Topic; the abnormal partition ID; the current replica count and the expected replica count; the timestamp when the alarm is triggered; relevant operation suggestions or emergency measures.
[0071] Furthermore, the alarm information can be sent by sending an alarm email to the operation and maintenance team or relevant responsible persons; sending the alarm information by text message to ensure that key personnel can receive the alarm in time; sending alarm notifications through instant messaging platforms such as enterprise WeChat, DingTalk, Slack, etc.; in the case of severe alarms, notifying relevant persons by automatically making phone calls.
[0072] S20: Determine the Brokers required to store the replicas corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each of the first partitions under the current topic, so as to obtain the first replica distribution list corresponding to the first partition.
[0073] In some embodiments of the present application, determining the Broker required to store the replica corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each first partition under the current theme to obtain the first replica distribution list corresponding to the first partition includes:
[0074] Obtain a Broker set of the Kafka cluster, and obtain all first Brokers storing replicas corresponding to the first partition from the real-time replica distribution list;
[0075] Filter out all the first Brokers from the Broker set, and obtain all the second Brokers that do not store a copy of the first partition;
[0076] According to the first preset threshold, several third Brokers are selected from all the second Brokers, and the first replica distribution list is generated according to all the third Brokers and all the first Brokers.
[0077] First, according to the copy distribution in the partition with the current abnormal number of copies, the second Broker that does not store the copy of the abnormal partition is obtained, and then a certain number of third Brokers are selected from all the second Brokers, so that after the final copy is stored in the third Broker, the total number of copies of the abnormal partition, that is, the sum of the first Broker and the third Broker, meets the first preset threshold, thereby automatically screening out the first copy distribution list for subsequent storage of copies, thereby improving the degree of automation of the Kafka cluster copy number exception handling process.
[0078] Preferably, in some embodiments of the present application, the first replica distribution list may also be obtained by the following method:
[0079] S21: Get the Broker that can store the current partition, and directly generate the first replica distribution list based on all Brokers;
[0080] S22: Automatically call the kafka-reassign-partitions.sh tool to reallocate replicas according to the first replica distribution list; specifically: ". / bin / kafka-reassign-partitions.sh--zookeeper192.168.62.32:2181--topics-to-move-json-filea.json--broker-list"0,1,2"–generate"; where the content of the a.json document is: {"topics":[{"topic":"shen"}],"version":1}; the first replica distribution list is: broker-list"0,1,2".
[0081] In order to more vividly illustrate the partitioning situation obtained through S22, the following Json document information is used for explanation:
[0082] {"version":1,"partitions":[{"topic":"shen","partition":3,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"shen","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"shen","partition":2,"replicas":[0,1],"log_dirs":["any","any"]},{"topic":"shen","partition":4,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"shen","partition":1,"replicas":[2,0],"log_dirs":["any","any"]}]}; Among them, the replicas of each partition are evenly distributed among Broker0 to Broker2, so that the load of each Broker is balanced.
[0083] In some embodiments of the present application, selecting a plurality of third Brokers from all the second Brokers according to the first preset threshold includes:
[0084] Sort all the second Brokers in ascending order according to the number of stored replicas in the second Broker;
[0085] A plurality of third Brokers are selected in sequence from all the sorted second Brokers until the sum of the number of the first Brokers and the number of the third Brokers is equal to a first preset threshold.
[0086] By giving priority to Brokers with lower loads, excessive pressure on a single Broker can be avoided during the replica replication process, thereby improving the stability of the system during the abnormal handling of the number of replicas in the Kafka cluster.
[0087] S30: According to the first replica distribution list of each first partition under the current topic and the business performance index of the current system, dynamically store the replica of each first partition in the corresponding Broker in sequence.
[0088] In some embodiments of the present application, dynamically storing the replica corresponding to the first partition in the corresponding Broker according to the first replica distribution list of each first partition and the service performance index of the current system includes:
[0089] Obtaining business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.;
[0090] Determining whether the current service performance indicator exceeds a second preset threshold;
[0091] If the business performance indicator exceeds the second preset threshold, the copy storage operation of the current first partition is stopped; otherwise, the copy of the current first partition is stored in each Broker in the first copy distribution list in sequence.
[0092] During the process of handling the abnormal number of replicas in the Kafka cluster, the process of copying the replicas to the Broker will affect the stability of data transmission in the business system, resulting in a decline in user experience and further affecting the business performance indicators of the business system. Therefore, when the business performance indicators fluctuate too much during the process of handling the abnormal number of replicas in the Kafka cluster, the stability of the system can be improved by temporarily pausing the replica copy operation and waiting for the business performance indicators to stabilize before continuing the copy operation.
[0093] Preferably, in some embodiments of the present application, the above step S30 can be performed by the following preferred implementation:
[0094] S31: Gradually add replicas to each abnormal partition to avoid large-scale changes to all partitions at once, thereby reducing the impact on the system;
[0095] S32: When selecting a new replica location, give priority to Brokers with lower current loads to avoid excessive pressure on a single Broker;
[0096] S33: When redistributing replicas, use rolling restart to update Brokers one by one to ensure that only a small number of Brokers are in the restart state at any point in time to avoid business interruption;
[0097] S34: During the process of adjusting the number of replicas, monitor business performance indicators in real time, such as message delay and consumption rate. Once an abnormality is found, suspend the adjustment operation immediately and continue after the system stabilizes.
[0098] Preferably, in some embodiments of the present application, after executing step S30, a recovery alarm is further generated by the monitoring system, and when the API of the monitoring system obtains the recovery alarm, it is confirmed that the alarm has been restored, specifically:
[0099] S35: When the monitoring system detects that the number of copies of the Kafka Topic has recovered to the preset threshold, the monitoring system automatically triggers a recovery alarm;
[0100] S36: The monitoring system generates a recovery alarm corresponding to the previous abnormal warning. The recovery alarm records which topics and partitions have recovered to normal numbers of replicas.
[0101] S37: Send a recovery notification to the operation and maintenance team or relevant person in charge to inform them that the system has returned to normal, and record the detailed information of the recovery alarm in the log, including the time of recovery, affected topics and partitions, etc., for subsequent analysis and auditing; after confirming that the alarm has been restored, continue to perform regular monitoring of the Kafka cluster to ensure the continued stable operation of the system.
[0102] In summary, the real-time control method of the number of copies of a Kafka cluster provided by the embodiment of the present application has the following beneficial effects compared to the prior art: by automatically and in real time obtaining the copy distribution of each partition under the current topic, accurately identifying the abnormal partitions with insufficient number of copies, and then automatically screening the Brokers required to store the newly added copies by pre-setting the number of normal copies and combining the copy distribution of the current abnormal partition, thereby improving the efficiency of recovering the number of copies of the abnormal partition and reducing the impact of manual intervention on the stability of the system. At the same time, since the system state changes during the process of recovering the number of copies, the system becomes unstable, thereby affecting the business. Therefore, in the process of recovering the number of copies in the abnormal partition according to the determined Broker, different measures are taken according to different system states by monitoring the business performance indicators of the system in real time, thereby reducing the negative impact on the business.
[0103] Embodiment 2
[0104] refer to Figure 2, a real-time control device for the number of copies of a Kafka cluster provided in an embodiment of the present application, comprising: a first partition acquisition module 11, a first copy distribution list acquisition module 12 and a copy dynamic storage module 13; wherein, the first partition acquisition module 11 is used to obtain all first partitions with abnormal copy numbers under the current topic according to the real-time copy distribution list of each partition under the current topic; the first copy distribution list acquisition module 12 is used to determine the Broker required to store the copy corresponding to the first partition according to a first preset threshold and the real-time copy distribution list of each first partition under the current topic, so as to obtain the first copy distribution list corresponding to the first partition; the copy dynamic storage module 13 is used to dynamically store the copies of each first partition in the corresponding Broker in sequence according to the first copy distribution list of each first partition under the current topic and the business performance indicators of the current system.
[0105] In some embodiments of the present application, the first replica distribution list acquisition module 12 is used to determine the Broker required to store the replica corresponding to the first partition based on the first preset threshold and the real-time replica distribution list of each first partition under the current topic, so as to obtain the first replica distribution list corresponding to the first partition, including: obtaining the Broker set of the Kafka cluster, and obtaining all the first Brokers that store the replicas corresponding to the first partition from the real-time replica distribution list; filtering out all the first Brokers from the Broker set, and obtaining all the second Brokers that do not store the replicas of the first partition; according to the first preset threshold, selecting a number of third Brokers from all the second Brokers, and generating the first replica distribution list based on all the third Brokers and all the first Brokers.
[0106] In some embodiments of the present application, selecting a number of third Brokers from all the second Brokers according to the first preset threshold includes: sorting all the second Brokers in ascending order according to the number of replicas stored in the second Brokers; and selecting a number of third Brokers from all the sorted second Brokers in turn until the sum of the number of the first Brokers and the third Brokers is equal to the first preset threshold.
[0107] In some embodiments of the present application, the replica dynamic storage module 13 is used to dynamically store the replicas of each first partition in the corresponding Broker in sequence according to the first replica distribution list of each first partition under the current topic and the business performance indicators of the current system, including: obtaining the business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.; judging whether the current business performance indicators exceed the second preset threshold; if the business performance indicators exceed the second preset threshold, stopping the replica storage operation of the current first partition; otherwise, storing the replica of the current first partition in sequence in each Broker in the first replica distribution list.
[0108] In some embodiments of the present application, the first partition acquisition module 11 is used to obtain all first partitions with abnormal number of copies under the current theme according to the real-time copy distribution list of each partition under the current theme, including: traversing the real-time copy distribution list of each partition under the current theme, and obtaining the first number of Brokers storing copies of each partition; when the first number corresponding to the partition is less than the first preset threshold, determining that the partition is the first partition.
[0109] In summary, the real-time control device for the number of copies of a Kafka cluster provided by the embodiment of the present application has the following beneficial effects compared to the prior art: by automatically and in real time acquiring the copy distribution of each partition under the current topic, accurately identifying abnormal partitions with insufficient number of copies, and then automatically screening the Brokers required to store newly added copies by pre-setting the number of normal copies and combining the copy distribution of the current abnormal partition, thereby improving the efficiency of recovering the number of copies of the abnormal partition and reducing the impact of manual intervention on the stability of the system. At the same time, since the system state changes during the process of recovering the number of copies, the system becomes unstable, thereby affecting the business. Therefore, in the process of recovering the number of copies in the abnormal partition according to the determined Broker, different measures are taken according to different system states by monitoring the business performance indicators of the system in real time, thereby reducing the negative impact on the business.
[0110] Embodiment 3
[0111] Based on the above-mentioned embodiment of the method for real-time control of the number of replicas of the Kafka cluster, refer to Figure 3Another embodiment of the present application provides a real-time control system for the number of Kafka cluster copies, the system comprising: a Kafka cluster, wherein the Kafka cluster includes several Kafka Brokers, which are responsible for data storage and transmission; a monitoring system built on the bottom layer of Zabbix, which is responsible for monitoring the status of the Kafka cluster and automatically calling the kafka-topics.sh tool to monitor the number of Kafka cluster copies, the monitoring system also includes several triggers and an alarm system; wherein the trigger is used to automatically determine whether the number of Kafka cluster copies obtained by monitoring is lower than a preset threshold, and when it is lower than or higher than the preset threshold, the corresponding alarm is automatically triggered; the alarm system is used to trigger the corresponding alarm action according to the alarm signal issued by the trigger; the recovery system realizes the dynamic recovery of the number of Kafka Topic copies by calling the API of the monitoring system.
[0112] In some embodiments of the present application, the recovery system can restore the number of Kafka Topic copies through the real-time control method of the number of Kafka cluster copies in Example 1.
[0113] Preferably, in some embodiments of the present application, the recovery system can also restore the number of Kafka Topic copies by the following preferred implementation method:
[0114] Step 1: Get alarm information regularly (call monitoring system API): Java project calls monitoring system API regularly to get alarm information;
[0115] Step 2: Handle the replica missing alarm (kafka-reassign): Call the kafka-reassign-partitions.sh tool to dynamically increase the number of replicas. During the process of increasing the number of replicas, ensure that the business is not affected.
[0116] Step 3: Confirm alarm recovery (call the monitoring system API): When the number of Kafka topic copies returns to the monitoring threshold, the monitoring system will generate a recovery alarm. After the Java project calls the monitoring system API to obtain the recovery alarm, it confirms that the alarm has been restored.
[0117] In summary, it can be seen that the real-time control system for the number of Kafka cluster copies provided by the embodiment of the present application has the following beneficial effects compared with the prior art: no human intervention is required, the system automatically completes monitoring and recovery operations, greatly reducing operation and maintenance costs and error rates; through continuous monitoring, it can quickly discover and handle abnormal situations in the number of copies, thereby improving the reliability and availability of the system.
[0118] Embodiment 4
[0119] Based on the above-mentioned embodiment of the real-time control method for the number of Kafka cluster copies, another embodiment of the present application provides a terminal device for real-time control of the number of Kafka cluster copies. The terminal device for real-time control of the number of Kafka cluster copies includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the real-time control method for the number of Kafka cluster copies of any embodiment of the present application is implemented.
[0120] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the real-time control device for the number of replicas of the Kafka cluster.
[0121] The real-time control device for the number of Kafka cluster copies can be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The real-time control terminal device for the number of Kafka cluster copies can include, but is not limited to, a processor and a memory.
[0122] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the real-time control device for the number of copies of the Kafka cluster, and uses various interfaces and lines to connect the various parts of the real-time control device for the number of copies of the entire Kafka cluster. The memory may be used to store the computer program and / or module, and the processor implements various functions of the real-time control device for the number of copies of the Kafka cluster by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0123] Embodiment 5
[0124] Based on the above-mentioned embodiment of the real-time control method for the number of Kafka cluster copies, another embodiment of the present application provides a storage medium, wherein the storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the real-time control method for the number of Kafka cluster copies of any embodiment of the present application.
[0125] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0126] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims
1. A real-time control method for the number of Kafka cluster replicas, characterized in that: include: According to the real-time replica distribution list of each partition under the current topic, obtain all first partitions with abnormal number of replicas under the current topic; Determine, according to a first preset threshold and a real-time replica distribution list of each of the first partitions under the current topic, a Broker required to store the replica corresponding to the first partition, so as to obtain a first replica distribution list corresponding to the first partition; According to the distribution list of the first copies of the first partitions under the current topic and the business performance indicators of the current system, the copies of the first partitions are dynamically stored in the corresponding Broker in sequence.
2. A method for real-time control of the number of copies of a Kafka cluster as described in claim 1, characterized in that: The determining, according to the first preset threshold and the real-time replica distribution list of each of the first partitions under the current theme, a Broker required to store the replica corresponding to the first partition to obtain the first replica distribution list corresponding to the first partition includes: Obtain a Broker set of the Kafka cluster, and obtain all first Brokers storing replicas corresponding to the first partition from the real-time replica distribution list; Filter out all the first Brokers from the Broker set, and obtain all the second Brokers that do not store a copy of the first partition; According to the first preset threshold, several third Brokers are selected from all the second Brokers, and the first replica distribution list is generated according to all the third Brokers.
3. A real-time control method for the number of Kafka cluster copies as described in claim 2, characterized in that: The selecting a plurality of third Brokers from all the second Brokers according to the first preset threshold comprises: Sort all the second Brokers in ascending order according to the number of stored replicas in the second Broker; A plurality of third Brokers are selected in sequence from all the sorted second Brokers until the sum of the number of the first Brokers and the number of the third Brokers is equal to a first preset threshold.
4. A real-time control method for the number of Kafka cluster copies as described in claim 1, characterized in that: The dynamically storing the replica corresponding to the first partition in the corresponding Broker according to the first replica distribution list of each first partition and the business performance index of the current system includes: Obtaining business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.; Determining whether the current service performance indicator exceeds a second preset threshold; If the business performance indicator exceeds the second preset threshold, the copy storage operation of the current first partition is stopped; otherwise, the copy of the current first partition is stored in each Broker in the first copy distribution list in sequence.
5. A real-time control method for the number of Kafka cluster copies as described in claim 1, characterized in that: The step of obtaining all first partitions with abnormal number of copies under the current topic according to the real-time copy distribution list of each partition under the current topic includes: Traverse the real-time replica distribution list of each partition under the current topic, and obtain the first number of Brokers storing replicas of each partition; When the first number corresponding to the partition is less than the first preset threshold, the partition is determined to be the first partition.
6. A real-time control device for the number of copies of a Kafka cluster, characterized in that: include: A first partition acquisition module, a first replica distribution list acquisition module, and a replica dynamic storage module; The first partition acquisition module is used to acquire all first partitions with abnormal number of copies under the current topic according to the real-time copy distribution list of each partition under the current topic; The first replica distribution list acquisition module is used to determine the Broker required to store the replica corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each first partition under the current theme, so as to obtain the first replica distribution list corresponding to the first partition; The replica dynamic storage module is used to dynamically store the replicas of each first partition in the corresponding Broker in sequence according to the first replica distribution list of each first partition under the current theme and the business performance indicators of the current system.
7. A Kafka cluster replica number real-time control device as claimed in claim 6, characterized in that: The first replica distribution list acquisition module is used to determine the Broker required to store the replica corresponding to the first partition according to the first preset threshold and the real-time replica distribution list of each first partition under the current theme, so as to obtain the first replica distribution list corresponding to the first partition, including: Obtain a Broker set of the Kafka cluster, and obtain all first Brokers storing replicas corresponding to the first partition from the real-time replica distribution list; Filter out all the first Brokers from the Broker set, and obtain all the second Brokers that do not store a copy of the first partition; According to the first preset threshold, several third Brokers are selected from all the second Brokers, and the first replica distribution list is generated according to all the third Brokers.
8. A Kafka cluster replica number real-time control device as claimed in claim 7, characterized in that: The selecting a plurality of third Brokers from all the second Brokers according to the first preset threshold comprises: Sort all the second Brokers in ascending order according to the number of stored replicas in the second Broker; A plurality of third Brokers are selected in sequence from all the sorted second Brokers until the sum of the number of the first Brokers and the number of the third Brokers is equal to a first preset threshold.
9. A Kafka cluster replica number real-time control device as claimed in claim 6, characterized in that: The replica dynamic storage module is used to dynamically store the replicas of each first partition in the corresponding Broker in sequence according to the first replica distribution list of each first partition under the current theme and the business performance index of the current system, including: Obtaining business performance indicators of the current business system; the business performance indicators include: message delay rate, consumption rate, etc.; Determining whether the current service performance indicator exceeds a second preset threshold; If the business performance indicator exceeds the second preset threshold, the copy storage operation of the current first partition is stopped; otherwise, the copy of the current first partition is stored in each Broker in the first copy distribution list in sequence.
10. A Kafka cluster replica number real-time control device as claimed in claim 6, characterized in that: The first partition acquisition module is used to acquire all first partitions with abnormal number of copies under the current topic according to the real-time copy distribution list of each partition under the current topic, including: Traverse the real-time replica distribution list of each partition under the current topic, and obtain the first number of Brokers storing replicas of each partition; When the first number corresponding to the partition is less than the first preset threshold, the partition is determined to be the first partition.
Citation Information
Cited By
Disk balancing method and device based on distributed message subscription platform
CN119937902A
A disk balancing method and device based on a distributed message subscription platform
CN119937902B
