A method and device for monitoring the operating status of a Kafka cluster

Abstract

The present invention discloses a method and apparatus for monitoring the operating status of a Kafka cluster. The method comprises: obtaining cluster data across multiple dimensions within the Kafka cluster based on the Kafka cluster's JMX port; aggregating the cluster data across multiple dimensions based on preconfigured monitoring items to generate aggregated data for the monitoring items; and monitoring the operating status of the Kafka cluster based on the aggregated data for the monitoring items. The present invention provides real-time monitoring of the operating status of a Kafka cluster, offering significant flexibility.

Description

Technical Field

[0001] The present application relates to the field of cluster management technology, and in particular to a method and device for monitoring the operating status of a Kafka cluster. Background Art

[0002] Currently, there are a variety of open-source message queue middleware solutions, such as RabbitMQ, RocketMQ, and Kafka, each of which is widely used. Kafka's popularity stems from its disk storage, strong backlog capacity, high stability, and convenient backdated consumption. Kafka can encounter various issues during operation, requiring users to promptly identify and prevent serious consequences. These operational metrics can be obtained through the JMX port.

[0003] However, Kafka provides a vast array of operational metrics, each categorized in a complex manner. Implementing a Kafka cluster monitoring and alerting system can lead to difficulties in selecting key metrics and categorizing them for unified data collection. Without categorizing Kafka's numerous monitoring metrics and their aggregation calculation methods, expanding new monitoring projects requires rewriting new monitoring code, significantly limiting the flexibility of the monitoring system. Summary of the Invention

[0004] The present application provides a method and device for monitoring the operating status of a Kafka cluster to solve the problem in the prior art that when expanding new monitoring projects, new monitoring codes are intelligently written, which greatly limits the flexibility of the monitoring system.

[0005] In order to solve the above problems, the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for monitoring the operating status of a Kafka cluster, comprising:

[0007] Based on the JMX port of the Kafka cluster, obtain cluster data in multiple dimensions within the Kafka cluster;

[0008] According to the pre-configured monitoring items, the cluster data under the multiple dimensions are aggregated to generate aggregated data of the monitoring items;

[0009] Monitor the running status of the Kafka cluster based on the aggregated data of the monitoring items.

[0010] Optionally, the multiple dimensions include: server node dimension, topic dimension, partition dimension and master partition dimension.

[0011] The JMX port based on the Kafka cluster obtains cluster data in multiple dimensions within the Kafka cluster, including:

[0012] Based on the JMX port, obtain node indicator data of the Kafka cluster under the server node dimension;

[0013] Obtain topic indicator data for each topic on the server node of the Kafka cluster under the topic dimension;

[0014] Obtain partition indicator data for each partition of each topic on the server node of the Kafka cluster under the partition dimension;

[0015] Obtain primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension.

[0016] Optionally, aggregating the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items includes:

[0017] When the monitoring item is a regular monitoring item, aggregating the cluster data under the multiple dimensions to generate aggregated data of the regular monitoring item;

[0018] When the monitoring items are all partition monitoring items, the partition indicator data of each partition of each topic are summed and aggregated to generate aggregated data of all partition monitoring items;

[0019] When the monitoring item is a node theme monitoring item, the partition indicator data of all themes of each server node are summed up and aggregated to generate aggregated data of the node theme monitoring item;

[0020] When the monitoring item is a monitoring item for all nodes, the node indicator data of all server nodes are aggregated and processed to generate aggregated data of the monitoring items for all nodes;

[0021] When the monitoring item is a subject monitoring item, the partition indicator data belonging to all topics are aggregated to generate aggregated data of the subject monitoring item;

[0022] When the monitoring item is a node sum monitoring item, the node indicator data of all server nodes are aggregated and summed to generate aggregated data of the node sum monitoring item.

[0023] Optionally, after aggregating the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items, the method further includes:

[0024] The aggregated data is saved in the storage space corresponding to the monitoring item.

[0025] Optionally, monitoring the operating status of the Kafka cluster based on the aggregated data of the monitoring items includes:

[0026] Based on the aggregated data, obtain the consumption displacement information and high watermark information corresponding to the Kafka cluster;

[0027] Determine message backlog information of the Kafka cluster based on the consumption displacement information and the high watermark information;

[0028] Determine the running status of the Kafka cluster based on the message backlog information.

[0029] In a second aspect, an embodiment of the present application provides a device for monitoring the operating status of a Kafka cluster, including:

[0030] The cluster data acquisition module is used to obtain cluster data in multiple dimensions within the Kafka cluster based on the JMX port of the Kafka cluster;

[0031] An aggregate data generation module is used to aggregate the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregate data of the monitoring items;

[0032] The operation status monitoring module is used to monitor the operation status of the Kafka cluster based on the aggregated data of the monitoring items.

[0033] Optionally, the multiple dimensions include: server node dimension, topic dimension, partition dimension and master partition dimension.

[0034] The cluster data acquisition module includes:

[0035] A first indicator data acquisition unit is configured to acquire node indicator data of the Kafka cluster under the server node dimension based on the JMX port;

[0036] A second indicator data acquisition unit is used to obtain topic indicator data of each topic on the server node of the Kafka cluster under the topic dimension;

[0037] A third indicator data acquisition unit is used to obtain partition indicator data of each partition of each topic on the server node of the Kafka cluster under the partition dimension;

[0038] The fourth indicator data acquisition unit is used to obtain the primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension.

[0039] Optionally, the aggregated data generation module includes:

[0040] A first aggregated data generating unit is configured to, when the monitoring item is a regular monitoring item, aggregate the cluster data under the multiple dimensions to generate aggregated data of the regular monitoring item;

[0041] A second aggregated data generating unit is configured to, when the monitoring item is a monitoring item of all partitions, perform summing and aggregation processing on the partition indicator data of each partition of each topic to generate aggregated data of the monitoring items of all partitions;

[0042] A third aggregated data generating unit is configured to, when the monitoring item is a node theme monitoring item, sum and aggregate the partition indicator data of all themes of each server node to generate aggregated data of the node theme monitoring item;

[0043] a fourth aggregated data generating unit, configured to aggregate the node indicator data of all server nodes to generate aggregated data of all node monitoring items when the monitoring items are all node monitoring items;

[0044] A fifth aggregate data generating unit is configured to aggregate the partition indicator data belonging to all topics to generate aggregate data of the topic monitoring item when the monitoring item is a topic monitoring item;

[0045] The sixth aggregate data generating unit is used to aggregate and sum the node indicator data of all server nodes to generate aggregate data of the node sum monitoring item when the monitoring item is a node sum monitoring item.

[0046] Optionally, the device further comprises:

[0047] The aggregate data saving module is used to save the aggregate data into the storage space corresponding to the monitoring item.

[0048] Optionally, the operating status monitoring module includes:

[0049] A displacement water level information acquisition unit, configured to acquire consumption displacement information and high water level information corresponding to the Kafka cluster based on the aggregated data;

[0050] A message backlog information determining unit, configured to determine the message backlog information of the Kafka cluster based on the consumption displacement information and the high watermark information;

[0051] A cluster operation status determination unit is used to determine the operation status of the Kafka cluster according to the message backlog information.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0053] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for monitoring the operating status of a Kafka cluster described in any one of the above items is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned methods for monitoring the operating status of a Kafka cluster.

[0055] Compared with the prior art, this application has the following advantages:

[0056] The embodiment of the present application provides a method and device for monitoring the operating status of a Kafka cluster. The method and device obtain cluster data under multiple dimensions within the Kafka cluster based on the JMX port of the Kafka cluster, aggregate the cluster data under multiple dimensions according to pre-configured monitoring items, generate aggregated data of the monitoring items, and monitor the operating status of the Kafka cluster based on the aggregated data of the monitoring items. The embodiment of the present application improves the classification of a series of monitoring indicators provided by the Kafka JMX port and summarizes the indicator aggregation method by monitoring type, so that flexible monitoring of various dimensions of the Kafka system can be achieved through simple configuration addition. Any dimension, any item, any calculation method can be specified, giving a very flexible configuration space for monitoring alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a method for monitoring the operating status of a Kafka cluster provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of the structure of a Kafka cluster operation status monitoring device provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0061] Example 1

[0062] Reference Figure 1 , shows a step flow chart of a method for monitoring the running status of a kafka cluster provided by an embodiment of the present application, such as Figure 1 As shown, the method for monitoring the running status of the Kafka cluster may include the following steps:

[0063] Step 101: Based on the JMX port of the Kafka cluster, cluster data in multiple dimensions within the Kafka cluster is obtained.

[0064] In this example, a Kafka cluster is a distributed messaging system written by LinkedIn in Scala. It serves as the foundation for LinkedIn's activity stream and operational data processing pipeline. It offers high scalability and throughput.

[0065] The JMX port refers to the port inside the Kafka cluster, which is mainly used to monitor the Kafka cluster.

[0066] Find the heap settings in the script kafka-server-start.sh that starts the Kafka cluster and add exportJMX_PORT="9999". The setting code is as follows:

[0067] if["x$KAFKA_HEAP_OPTS"="x"];

[0068] then export KAFKA_HEAP_OPTS="-Xmx1G-Xms1G"

[0069] export JMX_PORT="9999"

[0070] fi

[0071] Multiple dimensions refer to the dimensions used to collect data during the operation of the Kafka cluster. In this example, the multiple dimensions may include: server node dimension, topic dimension, partition dimension, and master partition dimension.

[0072] In practical applications, Kafka supports client languages: Kafka clients support most current mainstream languages, including C, C++, Erlang, Java, .NET, Perl, PHP, Python, Ruby, Go, and Javascript. You can use any of these languages to communicate with the Kafka server (i.e., you can write your own consumer to subscribe to messages from the Kafka cluster or write your own producer program).

[0073] The Kafka architecture consists of producer (message producer), consumer (message consumer), borker (the server of the Kafka cluster, responsible for processing message read and write requests and storing messages. At the Kafka cluster layer, there are actually many brokers), topic (message queue / classification is equivalent to a queue, which contains producer and consumer models), and zookeeper (metadata information is stored in zookeeper, including: storage consumption offset, topic information, partition information)).

[0074] Messages in Kafka are organized by topic. We can simply imagine this as a queue (in this example, a topic is called a topic), with each queue being a topic. Each topic is then divided into multiple partitions. This allows for parallelism. Within each partition, messages are strongly ordered, like an ordered queue. Each message has a sequence number offset, such as 0 to 12, and is read from the front and written to the back. Each partition corresponds to a broker, and a broker can manage multiple partitions. For example, if a topic has six partitions and two brokers, each broker would manage three partitions. A partition can be simply imagined as a file. When data is sent, it is appended to this partition. Messages are written directly to the file without being buffered in memory. Unlike many messaging systems, which delete messages after they are consumed, Kafka deletes messages based on a time policy, rather than upon consumption. In Kafka, there is no such thing as "consumed"; there is only the concept of "expired."

[0075] In actual applications, cluster data under multiple dimensions in the Kafka cluster can be obtained based on the JMS port of the Kafka cluster at preset time intervals. Specifically, the process of obtaining cluster data under multiple dimensions can be described in detail in combination with each dimension.

[0076] In a specific implementation of the present application, the multiple dimensions include: server node dimension, topic dimension, partition dimension, and master partition dimension. The above step 101 may include:

[0077] Sub-step A1: Based on the JMX port, obtain node indicator data of the Kafka cluster under the server node dimension.

[0078] In this embodiment, the node indicator data refers to the indicator data under the server node dimension.

[0079] When collecting Kafka cluster data, you can obtain node metrics data for the Kafka cluster in the server node dimension through the JMX port. In actual applications, each server node has only one metric data.

[0080] Sub-step A2: Obtain topic indicator data for each topic on the server node of the Kafka cluster under the topic dimension.

[0081] Topic indicator data refers to the data generated in the Kafka cluster under the topic dimension.

[0082] When collecting Kafka cluster data, you can obtain the topic indicator data of each topic on the server node of the Kafka cluster under the topic dimension based on the JMX port. In this example, each topic on the broker has one indicator data.

[0083] Sub-step A3: Obtain partition indicator data for each partition of each topic on the server node of the Kafka cluster under the partition dimension.

[0084] Partition indicator data refers to the indicator data generated in the Kafka cluster under the partition dimension.

[0085] When collecting Kafka cluster data, you can use the JMX port to obtain partition metrics for each partition of each topic on the server node of the Kafka cluster under the partition dimension. In this example, each partition of each topic on the broker has a metric data.

[0086] Sub-step A4: Obtain primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension.

[0087] Primary partition indicator data refers to the indicator data generated in the Kafka cluster under the topic partition dimension.

[0088] When collecting Kafka cluster data, you can use the JMX port to obtain the master partition metrics corresponding to the master partition in the partition metrics data of the Kafka cluster under the master partition dimension. In this example, each partition has only one master partition, namely leaderPartition.

[0089] After cluster data of multiple dimensions in the Kafka cluster is obtained based on the JMX port of the Kafka cluster, step 102 is executed.

[0090] Step 102: performing aggregation processing on the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items.

[0091] Monitoring items refer to pre-set items used to monitor the running status of the Kafka cluster. In specific implementations, monitoring items can include: general monitoring items, all partition monitoring items, node topic monitoring items, all node monitoring items, topic monitoring items, node sum monitoring items, etc.

[0092] In actual applications, business personnel can configure monitoring items according to their needs to better monitor the operating status of the Kafka cluster.

[0093] After obtaining cluster data in multiple dimensions within the Kafka cluster based on the JMX port of the Kafka cluster, the cluster data in multiple dimensions can be aggregated according to pre-configured monitoring items to generate aggregated data of the monitoring items.

[0094] The process of generating aggregated data of monitoring items may be described in detail in conjunction with the following embodiments.

[0095] In a specific implementation of the present application, step 102 may include:

[0096] Sub-step B1: When the monitoring item is a regular monitoring item, the cluster data under the multiple dimensions are aggregated to generate aggregated data of the regular monitoring item.

[0097] In this embodiment, when the monitoring item is a regular monitoring item, cluster data under multiple dimensions can be aggregated to generate unique data of the regular monitoring item, that is, the nomal monitoring item, and the collected metadata can be directly used.

[0098] Sub-step B2: When the monitoring item is a monitoring item for all partitions, the partition indicator data of each partition of each topic is summed and aggregated to generate aggregated data for the monitoring items for all partitions.

[0099] When the monitoring item is all partition monitoring items, the partition indicator data of each partition of each topic can be added and aggregated to generate aggregated data of all partition monitoring items, that is, the allPartitions monitoring item, which can summarize the collected data by topic - add the data of all partitions of a topic.

[0100] Sub-step B3: When the monitoring item is a node theme monitoring item, the partition index data of all themes of each server node are summed up and aggregated to generate aggregated data of the node theme monitoring item.

[0101] When the monitoring item is a node topic monitoring item, the partition indicator data of all topics of each server node can be added and summarized to generate aggregated data of the node topic monitoring item, that is, the brokerTopic monitoring item. The collected data can be summarized by all partitions of the topic on a broker - adding the data of all partitions of a topic on a broker.

[0102] Sub-step B4: When the monitoring item is a monitoring item for all nodes, the node indicator data of all server nodes are aggregated and processed to generate aggregated data of the monitoring items for all nodes.

[0103] When the monitoring item is the monitoring item of all nodes, the node indicator data of all server nodes can be aggregated to generate aggregated data of all node monitoring items, that is, brokerAll monitoring items, and the collected data can be aggregated by broker.

[0104] Sub-step B5: When the monitoring item is a subject monitoring item, the partition indicator data belonging to all subjects are aggregated to generate aggregated data of the subject monitoring item.

[0105] When the monitoring item is a topic monitoring item, the partition indicator data belonging to all topics can be aggregated to generate aggregated data of the topic monitoring item, that is, the clusterTopic monitoring item, and the collected data can be aggregated by topic.

[0106] Sub-step B6: When the monitoring item is a node sum monitoring item, the node indicator data of all server nodes are aggregated and summed to generate aggregated data of the node sum monitoring item.

[0107] When the monitoring item is a node sum monitoring item, the node indicator data of all server nodes can be summarized and processed to generate aggregate data of the node sum monitoring item, that is, the clusterAll monitoring item, which can summarize and add the collected broker data.

[0108] The embodiments of the present application improve the classification of a series of monitoring indicators provided by the Kafka JMX port and summarize the indicator aggregation method by monitoring type, so that flexible monitoring of various dimensions of the Kafka system can be achieved through simple configuration addition - any dimension, any project, and any calculation method can be specified, providing extremely flexible configuration space for monitoring alarms.

[0109] In this embodiment, after cluster data under multiple dimensions are aggregated according to pre-configured monitoring items to generate aggregate data of the monitoring items, the aggregate data can be saved in the storage space corresponding to the monitoring items. Specifically, storage spaces corresponding to different monitoring items can be pre-created in the system to classify and store the aggregate data corresponding to the monitoring items. Therefore, when business personnel retrieve the aggregate data of a specified monitoring item, they can directly search from the corresponding storage space, which can improve the search efficiency of the aggregate data and improve the monitoring efficiency of the operating status of the Kafka system.

[0110] After cluster data under multiple dimensions are aggregated according to pre-configured monitoring items to generate aggregated data of the monitoring items, step 103 is executed.

[0111] Step 103: Based on the aggregated data of the monitoring items, monitor the operating status of the Kafka cluster.

[0112] After aggregating the cluster data under multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items, the operating status of the Kafka cluster can be monitored based on the aggregated data of the monitoring items. Specifically, based on the aggregated data, the consumption displacement information and high water mark information corresponding to the Kafka cluster can be obtained, and based on the consumption displacement information and the high water mark information, the message backlog information of the Kafka cluster can be determined. Then, based on the message backlog information, the operating status of the Kafka cluster can be determined.

[0113] The embodiment of the present application provides a method for monitoring the running status of a Kafka cluster. The method obtains cluster data under multiple dimensions within the Kafka cluster through a JMX port based on the Kafka cluster, aggregates the cluster data under multiple dimensions according to pre-configured monitoring items, generates aggregated data of the monitoring items, and monitors the running status of the Kafka cluster based on the aggregated data of the monitoring items. The embodiment of the present application improves the classification of a series of monitoring indicators provided by the Kafka JMX port and summarizes the indicator aggregation method by monitoring type, so that flexible monitoring of various dimensions of the Kafka system can be achieved through simple configuration addition. Any dimension, any item, and any calculation method can be specified, giving a very flexible configuration space for monitoring alarms.

[0114] Example 2

[0115] Reference Figure 2 , shows a schematic diagram of the structure of a monitoring device for the running status of a Kafka cluster provided in an embodiment of the present application, such as Figure 2 As shown, the monitoring device 200 for the running status of the Kafka cluster may include the following modules:

[0116] The cluster data acquisition module 210 is used to acquire cluster data of multiple dimensions in the Kafka cluster based on the JMX port of the Kafka cluster;

[0117] Aggregate data generation module 220, configured to aggregate the cluster data under the multiple dimensions according to pre-configured monitoring items to generate aggregate data of the monitoring items;

[0118] The operation status monitoring module 230 is used to monitor the operation status of the Kafka cluster based on the aggregated data of the monitoring items.

[0119] Optionally, the multiple dimensions include: server node dimension, topic dimension, partition dimension and master partition dimension.

[0120] The cluster data acquisition module 210 includes:

[0121] A first indicator data acquisition unit is configured to acquire node indicator data of the Kafka cluster under the server node dimension based on the JMX port;

[0122] A second indicator data acquisition unit is used to obtain topic indicator data of each topic on the server node of the Kafka cluster under the topic dimension;

[0123] A third indicator data acquisition unit is used to obtain partition indicator data of each partition of each topic on the server node of the Kafka cluster under the partition dimension;

[0124] The fourth indicator data acquisition unit is used to obtain the primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension.

[0125] Optionally, the aggregated data generating module 220 includes:

[0126] A first aggregated data generating unit is configured to, when the monitoring item is a regular monitoring item, aggregate the cluster data under the multiple dimensions to generate aggregated data of the regular monitoring item;

[0127] A second aggregated data generating unit is configured to, when the monitoring item is a monitoring item of all partitions, perform summing and aggregation processing on the partition indicator data of each partition of each topic to generate aggregated data of the monitoring items of all partitions;

[0128] A third aggregated data generating unit is configured to, when the monitoring item is a node theme monitoring item, sum and aggregate the partition indicator data of all themes of each server node to generate aggregated data of the node theme monitoring item;

[0129] a fourth aggregated data generating unit, configured to aggregate the node indicator data of all server nodes to generate aggregated data of all node monitoring items when the monitoring items are all node monitoring items;

[0130] A fifth aggregate data generating unit is configured to aggregate the partition indicator data belonging to all topics to generate aggregate data of the topic monitoring item when the monitoring item is a topic monitoring item;

[0131] The sixth aggregate data generating unit is used to aggregate and sum the node indicator data of all server nodes to generate aggregate data of the node sum monitoring item when the monitoring item is a node sum monitoring item.

[0132] Optionally, the device further comprises:

[0133] The aggregate data saving module is used to save the aggregate data into the storage space corresponding to the monitoring item.

[0134] Optionally, the operating status monitoring module 230 includes:

[0135] A displacement water level information acquisition unit, configured to acquire consumption displacement information and high water level information corresponding to the Kafka cluster based on the aggregated data;

[0136] A message backlog information determining unit, configured to determine the message backlog information of the Kafka cluster based on the consumption displacement information and the high watermark information;

[0137] A cluster operation status determination unit is used to determine the operation status of the Kafka cluster according to the message backlog information.

[0138] The embodiment of the present application provides a monitoring device for the operation status of a Kafka cluster. The device obtains cluster data under multiple dimensions within the Kafka cluster through a JMX port based on the Kafka cluster, aggregates the cluster data under multiple dimensions according to pre-configured monitoring items, generates aggregated data of the monitoring items, and monitors the operation status of the Kafka cluster based on the aggregated data of the monitoring items. The embodiment of the present application improves the classification of a series of monitoring indicators provided by the Kafka JMX port and summarizes the indicator aggregation method by monitoring type, so that flexible monitoring of various dimensions of the Kafka system can be achieved through simple configuration addition. Any dimension, any item, any calculation method can be specified, giving a very flexible configuration space for monitoring alarms.

[0139] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0140] Example 3

[0141] An embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for monitoring the operating status of a Kafka cluster described in any one of the above items is implemented.

[0142] Reference Figure 3 , shows a schematic structural diagram of an electronic device provided by an embodiment of the present application, such as Figure 3 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 302 or computer program instructions loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0143] Multiple components in the electronic device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, a microphone, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0144] The various processes and procedures described above may be executed by the processing unit 301. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the CPU 301, one or more actions in the method described above may be performed.

[0145] Example 4

[0146] An embodiment of the present application provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the above-mentioned methods for monitoring the operating status of a Kafka cluster.

[0147] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0148] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, commodity, or device that includes the element.

[0149] The above is a detailed introduction to a method for monitoring the operating status of a Kafka cluster, a device for monitoring the operating status of a Kafka cluster, an electronic device and a storage medium provided in this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A method for monitoring the running status of a Kafka cluster, characterized in that: include: Based on the JMX port of the Kafka cluster, obtain cluster data in multiple dimensions within the Kafka cluster; According to the pre-configured monitoring items, the cluster data under the multiple dimensions are aggregated to generate aggregated data of the monitoring items; The multiple dimensions include: server node dimension, topic dimension, partition dimension and master partition dimension. The JMX port based on the Kafka cluster obtains cluster data under multiple dimensions in the Kafka cluster, including: Based on the JMX port, obtain node indicator data of the Kafka cluster under the server node dimension; Obtain topic indicator data for each topic on the server node of the Kafka cluster under the topic dimension; Obtain partition indicator data for each partition of each topic on the server node of the Kafka cluster under the partition dimension; Obtain primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension; The aggregating the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items includes: When the monitoring item is a regular monitoring item, aggregating the cluster data under the multiple dimensions to generate aggregated data of the regular monitoring item; When the monitoring items are all partition monitoring items, the partition indicator data of each partition of each topic are summed and aggregated to generate aggregated data of all partition monitoring items; When the monitoring item is a node theme monitoring item, the partition indicator data of all themes of each server node are summed up and aggregated to generate aggregated data of the node theme monitoring item; When the monitoring item is a monitoring item for all nodes, the node indicator data of all server nodes are aggregated and processed to generate aggregated data of the monitoring items for all nodes; When the monitoring item is a subject monitoring item, the partition indicator data belonging to all topics are aggregated to generate aggregated data of the subject monitoring item; When the monitoring item is a node sum monitoring item, the node indicator data of all server nodes are aggregated and summed to generate aggregated data of the node sum monitoring item; Monitor the running status of the Kafka cluster based on the aggregated data of the monitoring items.

2. The method according to claim 1, characterized in that After aggregating the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregated data of the monitoring items, the method further includes: The aggregated data is saved in the storage space corresponding to the monitoring item.

3. The method according to claim 1, characterized in that The monitoring of the running status of the Kafka cluster based on the aggregated data of the monitoring items includes: Based on the aggregated data, obtain the consumption displacement information and high watermark information corresponding to the Kafka cluster; Determine message backlog information of the Kafka cluster based on the consumption displacement information and the high watermark information; Determine the running status of the Kafka cluster based on the message backlog information.

4. A monitoring device for the operation status of a Kafka cluster, characterized in that: include: The cluster data acquisition module is used to obtain cluster data in multiple dimensions within the Kafka cluster based on the JMX port of the Kafka cluster; An aggregate data generation module is used to aggregate the cluster data under the multiple dimensions according to the pre-configured monitoring items to generate aggregate data of the monitoring items; An operation status monitoring module is used to monitor the operation status of the Kafka cluster based on the aggregated data of the monitoring items; The multiple dimensions include: server node dimension, subject dimension, partition dimension and master partition dimension, and the cluster data acquisition module includes: A first indicator data acquisition unit is configured to acquire node indicator data of the Kafka cluster under the server node dimension based on the JMX port; A second indicator data acquisition unit is used to obtain topic indicator data of each topic on the server node of the Kafka cluster under the topic dimension; A third indicator data acquisition unit is used to obtain partition indicator data of each partition of each topic on the server node of the Kafka cluster under the partition dimension; A fourth indicator data acquisition unit is used to obtain primary partition indicator data corresponding to the primary partition in the partition indicator data of the Kafka cluster under the primary partition dimension; The aggregated data generation module includes: A first aggregated data generating unit is configured to, when the monitoring item is a regular monitoring item, aggregate the cluster data under the multiple dimensions to generate aggregated data of the regular monitoring item; A second aggregated data generating unit is configured to, when the monitoring item is a monitoring item of all partitions, perform summing and aggregation processing on the partition indicator data of each partition of each topic to generate aggregated data of the monitoring items of all partitions; A third aggregated data generating unit is configured to, when the monitoring item is a node theme monitoring item, sum and aggregate the partition indicator data of all themes of each server node to generate aggregated data of the node theme monitoring item; a fourth aggregated data generating unit, configured to aggregate the node indicator data of all server nodes to generate aggregated data of all node monitoring items when the monitoring items are all node monitoring items; A fifth aggregate data generating unit is configured to aggregate the partition indicator data belonging to all topics to generate aggregate data of the topic monitoring item when the monitoring item is a topic monitoring item; The sixth aggregate data generating unit is used to aggregate and sum the node indicator data of all server nodes to generate aggregate data of the node sum monitoring item when the monitoring item is a node sum monitoring item.

5. The device according to claim 4, characterized in that The device further comprises: The aggregate data saving module is used to save the aggregate data into the storage space corresponding to the monitoring item.

6. The device according to claim 4, characterized in that The operating status monitoring module includes: A displacement water level information acquisition unit, configured to acquire consumption displacement information and high water level information corresponding to the Kafka cluster based on the aggregated data; A message backlog information determining unit, configured to determine the message backlog information of the Kafka cluster based on the consumption displacement information and the high watermark information; A cluster operation status determination unit is used to determine the operation status of the Kafka cluster according to the message backlog information.

7. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for monitoring the operating status of a Kafka cluster according to any one of claims 1 to 3 is implemented.

8. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method for monitoring the operating status of the Kafka cluster according to any one of claims 1 to 3.

Images