Distributed monitoring system based on Hadoop cluster and monitoring method thereof

Abstract

The invention discloses a distributed monitoring system based on a Hadoop cluster and a distributed monitoring method thereof. The system comprises a client, a master computer and a slave computer. The system is characterized in that: the master computer and the slave computer in the system are constructed based on a MapReduce framework of the Hadoop cluster; the client is used for submitting a monitoring work request to the master computer; after responding to the monitoring work request, the master computer divides monitoring work and distributes the divided monitoring work to the salve computer for independent completion; and the slave computer is used for completing an independent monitoring work task, integrating a result through the master computer and returning data to the client. By the invention, data such as the task schedule and the like of a distributed computing network MapReduce can be effectively monitored, the monitoring result can be precisely obtained in real time, and independent dynamic switching is realized under the condition of not influencing work execution.

Description

technical field [0001] The invention belongs to the technical field of distributed computing, and relates to a monitoring system applied to a Hadoop distributed operation system, in particular to a monitoring and control system for a large-scale distributed cluster running a MapReduce program. Background technique [0002] Hadoop is an open source distributed middleware for running MapReduce tasks. As an important technical component, it is widely used in various current cloud computing background systems. Generally, the number of commercial Hadoop clusters ranges from hundreds to thousands. As the scale of clusters managed by Hadoop becomes larger and larger, its management becomes more and more difficult. Generally, errors generated during Hadoop cluster operation can be divided into hardware errors, operating system errors, middleware errors, and user program errors according to their sources. There are many sources of errors, the interaction of various factors in the cl...

AI Technical Summary

Problems solved by technology

[0004] Existing monitoring systems have very obvious shortcomings in practical applications

First of all, the data obtained by the monitoring framework is not accurate enough, and there is a large amount of interference data. Whether using black-box or white-box methods, for a computer cluster with thousands of servers, the amount of monitored data is too large, even in self-learning Or with the help of classification algorithms, the analysis workload of these data is also very large

Secondly, the monitoring behavior cannot be started or stopped without changing the cluster state. In the actual production environment, the richer the debugging or monitoring behavior is, the greater the impact on system performance, and the behavior of the monitoring framework that cannot be dynamically changed cannot be well applied. production environment

In summary, there is still a lack of tools to effectively solve the above problems

Images