Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Big data platform monitoring method

A big data platform and data technology, applied in the computer field to ensure smooth completion, high sensitivity and reliability

Inactive Publication Date: 2018-10-26
电信科学技术第五研究所有限公司
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This design only monitors the system (CPU, memory, disk, big data platform components, etc.), function (computing jobs, data ETL, etc.) Data platform system process process alarm monitoring

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data platform monitoring method
  • Big data platform monitoring method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0014] see Figure 1~2 .

[0015] The system traffic referred to in the present invention refers to the total amount of data passing through a process, component, computing middleware, etc. at a certain time frequency, which is called the system traffic per unit time, referred to as the system traffic. For example, in the Extract process of data ETL, according to the time frequency per minute, the total amount of data passing through the Extract process is M, and the obtained system flow rate is M per minute, which can be calculated as M / m. Often according to the size of the data volume or the particularity of the business, the set time frequency is inconsistent.

[0016] Big data platform: A big data platform that needs to be monitored, including the basic data process, the final storage of data on the big data platform, and operations, analysis, and mining based on stored data.

[0017] Collection: The collection component includes component status collection and system tr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A big data platform monitoring method relates to the technology of computers. The invention comprises the following steps: 1) collecting state data from a big data platform, wherein the state data includes component state data; 2) calculating the collected result according to a predetermined rule, and storing the calculated result; and 3) if the calculation result satisfies the alarm condition, sending the alarm information through the API interface, wherein in the step 1), the state data further includes system traffic data, and the system traffic data includes system flows of various processes and components of the big data platform. The invention has the beneficial effects that the cluster, the nodes and the running condition of the computing operations on the nodes may be comprehensively and real-time monitored, the abnormality is detected and the alarm is timely, and the scheduling task is successfully completed, thereby having higher sensitivity and reliability.

Description

technical field [0001] The present invention relates to computer technology. Background technique [0002] For the current massive and ever-changing big data, storage is no longer the ultimate goal. How to obtain dividends, including commercial value, from the data is the real significance. By establishing a big data platform to obtain higher data value, this is the significance of data to enterprises. Although the ultimate task of the big data platform is not storage, data storage is the foundation. According to the particularity of the business, extract (transform)-transform (transform)-load (load) is often performed on the data before data storage. This process is called the data ETL process. Due to the diversity of businesses, other data processing processes, such as data cleaning, are often included before data is finally stored. [0003] Since more and more companies have built big data platforms, in order to improve the efficiency of platform work and the timelines...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08G06F11/30
CPCG06F11/3006G06F11/3055G06F11/3089H04L67/10
Inventor 俞鹏飞
Owner 电信科学技术第五研究所有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com