Monitoring system of large data platform

A technology of a big data platform and monitoring system, applied in the application field of computer and network technology, can solve the problems such as the inability to display the operation status of the big data platform in real time, the difficulty of the operation status of the big data platform, and the lack of a unified monitoring function.

Inactive Publication Date: 2015-05-13
BEIHANG UNIV
View PDF4 Cites 43 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Monitoring is an important part of the big data platform. The existing open source big data platforms lack an easy-to-use unified monitoring function. Specifically, it is difficult to obtain the run

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Monitoring system of large data platform
  • Monitoring system of large data platform
  • Monitoring system of large data platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0056] Such as figure 1 As shown, a big data platform monitoring system is built on the big data platform, including the collection layer, storage layer, function layer and display layer, the collection layer is responsible for collecting monitoring information in distributed clusters, and Stored in the database; the storage layer uses the relational database mysql, which is responsible for storing the monitoring data and statistical data collected by the collection layer; the functional layer is the service control end of the system, responsible for data processing and web interface request processing; the display layer is used to display Monitoring information, cluster management and job execution information, responsible for interacting with users.

[0057] Big data platform: The monitoring management system is built on the big data platform, which is composed of the Hadoop ecosystem, including the distributed file system HDFS, the resource management framework YARN, t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a monitoring system of a large data platform. The monitoring system of the large data platform comprises a large data platform operation information statistic module, a large data platform operation monitoring module and a large data platform operation statistic analysis module. The monitoring system of the large data platform resolves the following problems that firstly, the large data platform uses the Hadoop to store and manage data; secondly, for operation on the platform, the large data platform only stores the final state of operation, the middle state of operation is not recorded, and operation analysis is not facilitated; thirdly, the large data platform is lack of statistics and analysis of the operation state and trend, and only the current operation information can be obtained. The monitoring of assemblies in the platform is realized and displayed on an interface. The monitoring system of the large data platform realizes operation middle process monitoring, and collects and stores the input and output data amount and dependence information of operation. Through statistics and analysis of data in the operation process, the monitoring system of the large data platform realizes statistics and analysis of the operation trend in the large data platform.

Description

technical field [0001] The invention relates to the monitoring of resources and tasks of a big data platform, and belongs to the field of computer and network technology applications. Background technique [0002] With the continuous improvement of social information technology and the rapid popularization of Internet technology, the data to be processed is also increasing, and the demand for massive data processing in various fields is also increasing. Under the background that the storage space and computing power of a single machine can no longer meet people's demand for massive data processing, distributed computing and parallel computing began to develop and apply rapidly, and eventually developed into grid computing. The monitoring information of a large-scale distributed system is massive, and the monitoring resources are multi-level and multi-sourced. The dynamics and complexity of the big data platform bring many difficulties to the monitoring system of the big data...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/30G06F17/30
Inventor 熊桂喜乔少卿姜骁赵明杜博文
Owner BEIHANG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap