A monitoring system and monitoring method based on Kappa architecture

Abstract

The present invention provides a monitoring system and a monitoring method based on Kappa architecture, including a lower subsystem, a middle subsystem and an upper subsystem; the lower subsystem is composed of a data collector, a data communicator and a thread controller, and the contents involved include Zookeeper monitoring service, Netty client service and Future asynchronous callback technology; the middle subsystem is composed of a data receiver and a data classifier, and the contents involved include Netty server service, Zookeeper distributed coordination service and Kafka message engine; the upper subsystem is composed of a monitoring terminal and a data sensor, and the technologies involved include SpringBoot server and Flink stream processing technology. The application of this technical solution can alleviate the packet loss problem generated by the monitoring system developed based on the UDP protocol, and the transmission data packet redundancy problem generated by the monitoring system developed based on the HTTP protocol, and can also give play to the advantages of the Kappa architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of server monitoring systems, and in particular to a monitoring system and a monitoring method based on a Kappa architecture. Background Art

[0002] Kappa architecture is a real-time analysis stream processing architecture in the software engineering architecture. Its core parts mainly have two key points: stream computing engine and message engine. Compared with Lambda architecture, Kappa architecture has a simpler architecture, is stream data-centric, and has less development workload. It does not require additional development and maintenance of a huge ETL system and various intermediate report batch processing business systems. However, the system designed with Kappa architecture also has certain defects, that is, its functionality is not as comprehensive as Lambda architecture, and it is more suitable for processing various data sources with time as an important indicator.

[0003] In the stream processing computing business of the Kappa architecture, we generally consider three time indicators, namely event time, entry time, and processing time. Among them, event time refers to the specific time when an event occurs, entry time refers to the specific time when the event record flows to the message engine, and finally, processing time refers to the specific time when the event record is formally processed by the stream computing architecture. Most of the business of the monitoring system is related to event time, and a large amount of time series data can be used as an important indicator for business processing, so the Kappa architecture can be used as the system design architecture of the monitoring system.

[0004] At present, the transport layer protocols used by monitoring systems on the market mainly include TCP, UDP, and HTTP. However, except for TCP, UDP and HTTP have certain defects when used as transport layer protocols of monitoring systems: UDP is a connectionless and unreliable transport layer protocol. Its characteristics are that it cannot guarantee that the alarm message can reach the target host within the specified time under the condition of poor network environment and excessive load; HTTP contains relevant HTTP header information in each data packet. These header information are often redundant and useless in the monitoring stream data transmission business. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a monitoring system and a monitoring method based on the Kappa architecture, so as to alleviate the packet loss problem caused by the monitoring system developed based on the UDP protocol and the transmission data packet redundancy problem caused by the monitoring system developed based on the HTTP protocol, while also being able to give full play to the advantages of the Kappa architecture.

[0006] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a monitoring system based on Kappa architecture, comprising a lower subsystem, a middle subsystem and an upper subsystem; the lower subsystem is composed of a data collector, a data communicator and a thread controller, and the contents involved include Zookeeper monitoring service, Netty client service and Future asynchronous callback technology; the middle subsystem is composed of a data receiver and a data classifier, and the contents involved include Netty server service, Zookeeper distributed coordination service and Kafka message engine; the upper subsystem is composed of a monitoring terminal and a data sensor, and the technologies involved include SpringBoot server and Flink stream processing technology.

[0007] In a preferred embodiment, the data collector includes a configuration file IO module, a log file reading module and a local data management module;

[0008] The data communicator includes a Netty client module, a log file version cache module and a Zookeeper node tree module;

[0009] The thread controller includes a thread distribution module, a thread control module, a script driving module and a thread state rollback module.

[0010] In a preferred embodiment, the data receiver includes a Netty server module and a data preprocessing module;

[0011] The data classifier includes a data classification module and a Kafka publishing end module.

[0012] In a preferred embodiment, the data sensor includes a flow calculation node, a cluster performance calculation node, an indicator baseline alarm calculation node, and various specific business analysis nodes;

[0013] The monitoring terminal includes a Kafka message subscription terminal module, a view layer interface module, a cluster service scheduling module, a resource information management module and a monitoring terminal Zookeeper node tree module.

[0014] The present invention also provides a monitoring method based on the Kappa architecture, comprising the following steps:

[0015] Step 1: Monitoring data collection. The lower-level subsystem deployed on the monitored node will regularly collect the OS monitoring data, application log data, and business logic log data of the server, and send these data in JSON format through the Netty client to the middle-level subsystem located in the event routing server; at the same time, local business data directly from the business client is also transmitted to the middle-level subsystem through Cookie;

[0016] Step 2: Monitoring data preprocessing: The middle-level subsystem collects various monitoring stream data and data directly from the user client through the data preprocessing module. According to the strategy, it corrects the missing data and filters the useless data. At the same time, it collects the processed stream data into the local data classifier module.

[0017] Step 3: Monitor data classification. The data classifier distributes the data to different Kafka topics according to different categories, urgency, and specific functional requirements based on the data processing results and the types of source data JSON tags.

[0018] Step 4: Monitor data processing, use the Flink cluster managed by yarn to pull data from Kakfa, analyze various types of data in real time, generate view information, and generate notification information based on the pre-designed indicator baseline, and distribute different processing results to different Kafka topics;

[0019] Step 5: View layer data and notification data storage. For view layer data that can be directly passed to the view layer for display, the monitoring terminal developed using SpringBoot can obtain relevant data by listening to the specified KafkaTopic and store it in the local cache maintained by JVM. For notification data, if it is scheduling notification data, proceed to step 7 for processing. If it is other notification data, cache it in the persistent data layer.

[0020] Step 6: The data in the view layer shows that the front-end system of the monitoring system can obtain static data through the HTTP interface and obtain continuous streaming data through webscoket technology; at the same time, the monitoring terminal should also provide a control interface;

[0021] Step 7: Application service scheduling. For scheduling notification information, the monitoring terminal can obtain scheduling information by listening to the specified Kafkatopic. It can also obtain scheduling information from the front-end system and convert these data into corresponding operations on the Zookeeper node tree to change specific nodes. After the lower-level subsystem monitors the node changes, it will call the thread controller module to schedule the monitored application service.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention provides a monitoring system and a monitoring method based on the Kappa architecture. Starting from the underlying network transport layer protocol, the TCP protocol is used to avoid the packet loss problem of the UDP protocol and the redundant data problem of the HTTP protocol. At the same time, the Reactor reactor design pattern is combined, and the data transmission volume between servers can be increased by increasing the number of communication threads when the bandwidth permits. The important information of each server in the monitored cluster will be stored in Zookeeper, and the information can be redundantly stored in the cluster, eliminating the problem of memory data loss caused by single point failure. Due to the partition allocation algorithm of Kafka, data can be evenly distributed and stored in the message engine, which can prevent the occurrence of data skew to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a deployment design diagram of each subsystem of the preferred embodiment of the present invention;

[0025] Figure 2 An example table of lower-level subsystem function modules according to a preferred embodiment of the present invention;

[0026] Figure 3 An example table of middle-level subsystem function modules in a preferred embodiment of the present invention;

[0027] Figure 4 An example table of functional modules of the upper subsystem monitoring terminal system of the preferred embodiment of the present invention;

[0028] Figure 5 An example table of upper subsystem sensor function modules of a preferred embodiment of the present invention;

[0029] Figure 6 An example table of Zookeeper node design for a preferred embodiment of the present invention;

[0030] Figure 7 This is an example diagram of a data sensor stream processing topology according to a preferred embodiment of the present invention;

[0031] Figure 8 An example table for designing the storage of Kafka channel topic services in a preferred embodiment of the present invention;

[0032] Fig. 9 A flowchart of starting the lower subsystem of a preferred embodiment of the present invention;

[0033] Fig.10 This is a data interaction flow chart of a monitoring system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0037] A monitoring system based on Kappa architecture, reference Figures 1 to 10 , including the lower-level subsystem, the middle-level subsystem and the upper-level subsystem; the lower-level subsystem is composed of data collectors, data communicators and thread controllers, involving Zookeeper monitoring services, Netty client services and Future asynchronous callback technology; the middle-level subsystem is composed of data receivers and data classifiers, involving Netty server services, Zookeeper distributed coordination services and Kafka message engines; the upper-level subsystem is composed of monitoring terminals and data sensors, and the technologies involved include SpringBoot server and Flink stream processing technology.

[0038] The data collector includes a configuration file IO module, a log file reading module and a local data management module; the data communicator includes a Netty client module, a log file version cache module and a Zookeeper node tree module; the thread controller includes a thread distribution module, a thread control module, a script driver module and a thread state rollback module. The data receiver includes a Netty server module and a data preprocessing module; the data classifier includes a data classification module and a Kafka publishing module.

[0039] The data sensor includes a flow calculation node, a cluster performance calculation node, an indicator baseline alarm calculation node and various specific business analysis nodes; the monitoring terminal includes a Kafka message subscription terminal module, a view layer interface module, a cluster service scheduling module, a resource information management module and a monitoring terminal Zookeeper node tree module.

[0040] A monitoring method based on Kappa architecture includes the following steps:

[0041] Step 1: Monitoring data collection. The lower-level subsystem deployed on the monitored node will regularly collect the OS monitoring data, application log data, and business logic log data of the server, and send these data in JSON format through the Netty client to the middle-level subsystem located in the event routing server; at the same time, local business data directly from the business client is also transmitted to the middle-level subsystem through Cookie;

[0042] Step 2: Monitoring data preprocessing: The middle-level subsystem collects various monitoring stream data and data directly from the user client through the data preprocessing module. According to the strategy, it corrects the missing data and filters the useless data. At the same time, it collects the processed stream data into the local data classifier module.

[0043] Step 3: Monitor data classification. The data classifier distributes the data to different Kafka topics according to different categories, urgency, and specific functional requirements based on the data processing results and the types of source data JSON tags.

[0044] Step 4: Monitor data processing, use the Flink cluster managed by yarn to pull data from Kakfa, analyze various types of data in real time, generate view information, and generate notification information based on the pre-designed indicator baseline, and distribute different processing results to different Kafka topics;

[0045] Step 5: View layer data and notification data storage. For view layer data that can be directly passed to the view layer for display, the monitoring terminal developed using SpringBoot can obtain relevant data by listening to the specified KafkaTopic and store it in the local cache maintained by JVM. For notification data, if it is scheduling notification data, proceed to step 7 for processing. If it is other notification data, cache it in the persistent data layer.

[0046] Step 6: The data in the view layer shows that the front-end system of the monitoring system can obtain static data through the HTTP interface and obtain continuous streaming data through webscoket technology; at the same time, the monitoring terminal should also provide a control interface;

[0047] Step 7: Application service scheduling. For scheduling notification information, the monitoring terminal can obtain scheduling information by listening to the specified Kafkatopic. It can also obtain scheduling information from the front-end system and convert these data into corresponding operations on the Zookeeper node tree to change specific nodes. After the lower-level subsystem monitors the node changes, it will call the thread controller module to schedule the monitored application service.

[0048] The present invention is applied to the Netty framework, which uses the ByteBuf buffer that the Netty program carries to perform "real" read and write operations. The ByteBuf buffer is not located in the JVM heap, but is created by the ByteBufAllocator singleton class through the system call-related malloc() function, so the ByteBuf buffer is also called a direct buffer (Direct ByteBuf), and compared to the HeapByteBuf stored in the JVM heap, DirectByteBuf performs well in writing business. Considering that DirectByteBuf has a large creation and destruction overhead, and has the same memory overflow problem as HeapByteBuf, and the lower subsystem is mainly based on writing business, the present invention recommends that when applying to create a DirectByteBuf buffer, a pooling mode is adopted. The pooling allocator of the Netty program is managed by itself, and in a big data environment, memory overflow can be effectively avoided to a certain extent.

[0049] The present invention is applied to the Zookeeper distributed coordination system, and the registration information maintained by the system will all be cached in the memory of each Zookeeper server, and stored in a tree structure. Therefore, the official document of Zookeeper clearly mentions that the size of each node of the Zookeeper storage tree should not be 1M, and the height of the node tree should not be too large, otherwise it may affect the performance of the Zookeeper coordination service. Therefore, in actual development, the storage content of the Zookeeper node tree needs to be reasonably planned.

[0050] The present invention is applied to the Kafka message engine architecture. Since all actual read and write operations of Kafka are performed on the partitionLeader, it has its own intelligent partitionLeader election algorithm, which can distribute the partitionLeader to different physical nodes. This realizes the data load balancing service of the message engine layer and avoids the data skew problem to a certain extent. If other message engine systems are to be considered, it is necessary to ensure that the message engine at least has stream replay technology, otherwise the system developed based on the Kappa architecture will find it difficult to execute services related to offline computing.

[0051] Before executing step 1, you need to do the following work in sequence: ① Initialize each module of the lower-level subsystem. ② Initialize the connection between the Curator client and the Zookeeper server, and register the configuration information. The monitoring terminal can monitor the registration actions of the lower-level subsystem in a timely manner and update the local cache content of the monitoring terminal. ③ Initialize the thread controller, query the application service information that needs to be started from Zookeeper, and start the applications that need to be monitored in turn. ④ Initialize the working thread that monitors the Redis publish / subscribe channel. ⑤ Start the Netty client through the thread controller; After all the above steps are completed, the data collector can start working normally, collect the required monitoring data, and distribute it to the middle-level subsystem.

[0052] The startup service of the entire system also has prerequisites, that is, the Zookeeper cluster and the Kafka cluster must be started and run normally first, otherwise the information interaction between the three subsystems cannot be completed and the monitoring business cannot be implemented normally.

[0053] The present invention provides a monitoring system and a monitoring method based on the Kappa architecture. Starting from the underlying network transport layer protocol, the TCP protocol is used to avoid the packet loss problem of the UDP protocol and the redundant data problem of the HTTP protocol. At the same time, the Reactor reactor design pattern is combined, and the data transmission volume between servers can be increased by increasing the number of communication threads when the bandwidth permits. The important information of each server in the monitored cluster will be stored in Zookeeper, and the information can be redundantly stored in the cluster, eliminating the problem of memory data loss caused by single point failure. Due to the partition allocation algorithm of Kafka, data can be evenly distributed and stored in the message engine, which can prevent the occurrence of data skew to a certain extent.

Claims

1. A monitoring method for a monitoring system based on a Kappa architecture, characterized in that: The monitoring system includes a lower subsystem, a middle subsystem and an upper subsystem; the lower subsystem consists of a data collector, a data communicator and a thread controller, and involves Zookeeper monitoring services, Netty client services and Future asynchronous callback technology; the middle subsystem consists of a data receiver and a data classifier, and involves Netty server services, Zookeeper distributed coordination services and Kafka message engines; the upper subsystem consists of a monitoring terminal and a data sensor, and involves technologies such as SpringBoot server and Flink stream processing technology; The data collector includes a configuration file IO module, a log file reading module and a local data management module; The data communicator includes a Netty client module, a log file version cache module and a Zookeeper node tree module; The thread controller includes a thread distribution module, a thread control module, a script driving module and a thread state rollback module; The data receiver includes a Netty server module and a data preprocessing module; The data classifier includes a data classification module and a Kafka publishing end module; The data sensors include flow calculation nodes, cluster performance calculation nodes, index baseline alarm calculation nodes and various specific business analysis nodes; The monitoring terminal includes a Kafka message subscription terminal module, a view layer interface module, a cluster service scheduling module, a resource information management module and a monitoring terminal Zookeeper node tree module; The following steps are involved: Step 1: Monitoring data collection. The lower-level subsystem deployed on the monitored node will regularly collect the OS monitoring data, application log data, and business logic log data of the server, and send these data in JSON format through the Netty client to the middle-level subsystem located in the event routing server; at the same time, local business data directly from the business client is also transmitted to the middle-level subsystem through Cookie; Step 2: Monitoring data preprocessing: The middle-level subsystem collects various monitoring stream data and data directly from the user client through the data preprocessing module. According to the strategy, it corrects the missing data and filters the useless data. At the same time, it collects the processed stream data into the local data classifier module. Step 3: Monitor data classification. The data classifier distributes the data to different Kafka topics according to different categories, urgency, and specific functional requirements based on the data processing results and the types of source data JSON tags. Step 4: Monitor data processing. Use the Flink cluster managed by yarn to pull data from Kafka, analyze various types of data in real time, generate view information, and generate notification information based on the pre-designed indicator baseline. Distribute different processing results to different Kafka topics. Step 5: View layer data and notification data storage. For view layer data directly transmitted to the view layer for display, the monitoring terminal developed using SpringBoot can obtain relevant data by listening to the specified KafkaTopic and store it in the local cache maintained by JVM. For notification data, if it is scheduling notification data, proceed to step 7 for processing. If it is other notification data, cache it in the persistent data layer. Step 6: The data display of the view layer shows that the front-end system of the monitoring system obtains static data through the HTTP interface and obtains continuous streaming data through the websocket technology; at the same time, the monitoring terminal also provides a control interface; Step 7: Application service scheduling. For scheduling notification information, the monitoring terminal can obtain scheduling information by listening to the specified KafkaTopic. It can also obtain scheduling information from the front-end system and convert these data into corresponding operations on the Zookeeper node tree to change specific nodes. After the lower-level subsystem monitors the node changes, it will call the thread controller module to schedule the monitored application service.

Images