Real-time flow calculation monitoring system and method for processing monitoring big data

Abstract

The invention discloses a real-time stream computing monitoring system and method for processing monitoring big data, and the system comprises a data collection unit which employs a monitoring Agent,and obtains monitoring original data on a corresponding machine through an active reporting or active collection mode; The data caching unit comprises a high-performance message queue Kafka for storing the monitoring original data collected by the monitoring Agent and storing the monitoring data subjected to aggregation calculation; The data calculation unit is used for carrying out aggregation calculation on the original monitoring data based on a high concurrent flow calculation engine Flink, and the monitoring data subjected to aggregation calculation is sent to a high-performance message queue Kafka for temporary caching; The monitoring state management service unit is used for monitoring the operation states of the data acquisition unit and the data calculation unit; According to thesystem and the method disclosed by the invention, the accuracy of high-concurrency calculation is ensured, and the display speed is increased.

Description

technical field [0001] The invention relates to the technical field of big data computing, in particular to a real-time streaming computing monitoring system and method for processing and monitoring big data. Background technique [0002] With the increasing popularity of cloud computing big data technology, the rapid development of distributed technology, the increasingly complex scale of business systems, and the increasingly large scale of machines, the corresponding monitoring data has gradually become a kind of big data (hereinafter referred to as "monitoring big data"). The traditional monitoring system is unable to cope with monitoring big data, so there is an urgent need for a new monitoring system with high concurrent processing capabilities. [0003] As we all know, the value of data decreases rapidly with the passage of time, and the time requirement for monitoring big data is more urgent. If the system takes a long time to process the fault data after monitoring,...

AI Technical Summary

Problems solved by technology

Although stream computing technology is used, Spark Streaming is a "micro-batch" (Micro-Batch) technology, which is essentially batch processing, not real stream computing, and cannot process data based on monitoring data generation time (Event Time) , the error rate is relatively high

Images