Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data computing platform monitoring system and method

A computing platform and monitoring system technology, applied in the field of distributed computing, can solve problems such as lack of effective support, difficulty in starting and stopping of machines and services, submission and cancellation of jobs, and insufficient support for fine-grained monitoring of cluster computing resources. Use Responsive, Quick Response, Reduced Complexity Effects

Active Publication Date: 2019-11-12
NORTH CHINA UNIVERSITY OF TECHNOLOGY
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The second aspect: the current work does not support fine-grained monitoring of cluster computing resources.
The current work mostly monitors the job status or file system status of the Hadoop cluster, but there is no effective support for fine-grained monitoring of computing resources in the cluster, such as CPU, memory, bandwidth, and hard disk resources.
[0006] The third aspect: the current work does not support the service management of computing nodes
Current work rarely monitors the computing nodes in the cluster from the perspective of service management
[0007] The fourth aspect: the current work generally lacks graphical management interface and intelligent assistance support
This makes it difficult for business personnel who do not understand the installation of computing cluster environments to operate functions such as starting and stopping machines and services in the cluster, submitting and canceling jobs; for business personnel who do not understand development details, it is difficult to memorize various Parameters such as configuration and startup class names

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data computing platform monitoring system and method
  • Big data computing platform monitoring system and method
  • Big data computing platform monitoring system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0067] In a specific embodiment, the present invention is built on a big data computing platform and mainly includes three modules. combine figure 1 The big data computing platform and each module of the monitoring system of the big data computing platform are described in detail. in:

[0068] In a specific embodiment, the big data computing platform targeted by the present invention includes a Hadoop computing platform, a Storm computing platform, and a distributed cluster to which the computing platform belongs. The Hadoop computing platform is used for offline processing of massive data, and the Storm computing platform is used for online processing of real-time data. The distributed clusters to which both types of computing platforms belong consist of machines called computing nodes.

[0069] The invention includes a data acquisition module, which acquires and stores the monitoring indicators of the computing environment and the monitoring indicators of the computing jo...

Embodiment 2

[0075] In a specific embodiment, the monitoring system provided by the present invention can adopt the following preferred deployment methods, combining figure 2 Describe the deployment of each module and the interaction relationship between modules in the present invention. Regarding the deployment of the monitoring system of the big data computing platform, it specifically includes:

[0076] (1) Deploy big data computing platform: For Hadoop computing platform and Storm computing platform, each platform is deployed independently.

[0077] (2) Deploy the monitoring system of the big data computing platform: specifically include,

[0078] a) The agent program of the data acquisition module, a part is deployed on each computing node in the big data computing platform, and the other part is deployed on the monitoring server; the agent program of the data acquisition module deployed on the computing node, and the data acquisition module deployed on the monitoring server Module...

Embodiment 3

[0087] In a specific preferred embodiment, combined with image 3 Describes the process for submitting assignments. The job monitoring module encapsulates related command-line programs, and intelligent analysis makes it unnecessary for users to pay attention to different types of jobs and computing platforms, and does not need to memorize the configuration and many parameters of different types of computing platforms, which simplifies the complexity of users submitting jobs. The job monitoring module is responsible for receiving the computing job package (that is, the computing job program package) uploaded by the user, and submitting it after intelligent analysis. Specifically include the following steps:

[0088] (1) The user uploads the computing job package: the computing job package is in the jar file format, which is the program package of the batch computing job implemented by the user based on Hadoop programming, or the program package of the stream processing computi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a big data computing platform monitoring system and method. The system comprises a data collection module used for acquiring and storing monitoring indexes of a computing environment and monitoring indexes of computing operation in a computing platform in real time, and opening an query interface for an environment monitoring module and an operation monitoring module; the environment monitoring module used for monitoring the computing platform of the computing platform in real time, acquiring the monitoring indexes of the computing environment of the big data computing platform from the data collection module and visibly displaying these monitoring indexes of the computing environment; and the operation monitoring module used for monitoring the computing operation of the computing platform in real time, acquiring the monitoring indexes of the computing operation of the big data computing platform from the data collection module and visibly displaying the monitoring indexes of the computing operation. The big data computing platform monitoring system provided by the invention can reduce the complexity of the management operation procedures of the big data computing platform, and provides an intelligent and intuitive monitoring system that is easy to use and makes responses quickly.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a monitoring system for Storm and Hadoop big data computing platforms. Background technique [0002] With the continuous improvement of social information technology, many business fields generate massive and real-time data, and different fields have their own needs for data processing. At present, big data analysis is gradually dominated by unstructured data, and the storage space and computing power of a single machine can no longer be satisfied. Therefore, the Apache Hadoop and Apache Storm systems for massive data and real-time data calculation in a distributed environment are widely used in the business field. In application, it has gradually become the mainstream of the industry, and has also become the current de facto standard. [0003] The number of commercially available Hadoop and Storm clusters in the industry ranges from hundreds to thousands, which ma...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L12/26H04L29/08
CPCH04L43/08H04L67/10
Inventor 丁维龙赵卓峰曹娅琪胡雅鹏
Owner NORTH CHINA UNIVERSITY OF TECHNOLOGY