A method and device for data collection and monitoring

Abstract

The embodiment of the invention discloses a data acquisition monitoring method and device. The heartbeat data reported by a first type of slave server are acquired, wherein the heartbeat data include data volume information of files required to be acquired by the first type of slave server and the completely acquired data volume information of the files; and whether the first type of slave server fails to work for the acquisition process of the files is judged according to the data volume information of the files required to be acquired by the first type of slave server and the completely acquired data volume information of the files, and the data after the position of the failure in the files is the last completely acquired data of the files are located when the first type of slave server fails to work for the acquisition process of the files. The failure occurred in the distributed acquisition process of the business log data can be rapidly discovered and accurately located based on the heartbeat data.

Description

technical field [0001] The invention relates to the technical field of data acquisition, in particular to a data acquisition and monitoring method and device. Background technique [0002] With the development of the Internet, the business volume has increased rapidly, the scale of server clusters providing services has become larger and larger, and the types of services provided have become more and more complex. The reliability of business log data collection under the background of such big data is becoming more and more important. [0003] The current mainstream business log data collection system is a highly available, highly reliable, and distributed Flume system for massive log collection, aggregation, and transmission provided by Cloudera. At present, the business log data collection process based on the Flume system is as follows: First, the Flume system is used to collect business log data in a distributed manner, then the collected business log data is aggregated...

AI Technical Summary

Problems solved by technology

[0004] However, in the context of big data, the business log data that needs to be collected is distributed on many servers, and a single server has multiple business log data that need to be collected separately, resulting in a large number of collection tasks

In the process of distributed collection of business log data using the Flume system, due to the lack of reliable data collection and monitoring technology in the Flume system, faults that occur during the distributed collection of business log data cannot be quickly discovered and accurately located

Images