Real-time flow calculation monitoring system and method for processing monitoring big data

A technology for monitoring data and monitoring systems, applied in the field of big data computing, which can solve the problems of data processing and high error rate

Pending Publication Date: 2019-05-24
QINGDAO TELD NEW ENERGY TECH CO LTD
View PDF5 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although stream computing technology is used, Spark Streaming is a "micro-batch" (Micro-Batch) technology, which is essentially batch processin

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time flow calculation monitoring system and method for processing monitoring big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

[0042] The present invention provides a real-time streaming computing monitoring system and method for processing and monitoring big data, such as figure 1As shown, the system and method improve the performance of collecting and monitoring big data, calculate based on event time (Event Time) to ensure the accuracy of high concurrent calculation, and speed up the display speed by displaying aggregated data and being able to drill to the original data.

[0043] The specific implementation is as follows:

[0044] 1. Deploy the monitoring agent implemented by the patent of the present invention on each machine, and obtain the monitoring data on the corresponding machine through active reporting or active pulling.

[0045] The monitoring data format defined by the patent of the pr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a real-time stream computing monitoring system and method for processing monitoring big data, and the system comprises a data collection unit which employs a monitoring Agent,and obtains monitoring original data on a corresponding machine through an active reporting or active collection mode; The data caching unit comprises a high-performance message queue Kafka for storing the monitoring original data collected by the monitoring Agent and storing the monitoring data subjected to aggregation calculation; The data calculation unit is used for carrying out aggregation calculation on the original monitoring data based on a high concurrent flow calculation engine Flink, and the monitoring data subjected to aggregation calculation is sent to a high-performance message queue Kafka for temporary caching; The monitoring state management service unit is used for monitoring the operation states of the data acquisition unit and the data calculation unit; According to thesystem and the method disclosed by the invention, the accuracy of high-concurrency calculation is ensured, and the display speed is increased.

Description

technical field [0001] The invention relates to the technical field of big data computing, in particular to a real-time streaming computing monitoring system and method for processing and monitoring big data. Background technique [0002] With the increasing popularity of cloud computing big data technology, the rapid development of distributed technology, the increasingly complex scale of business systems, and the increasingly large scale of machines, the corresponding monitoring data has gradually become a kind of big data (hereinafter referred to as "monitoring big data"). The traditional monitoring system is unable to cope with monitoring big data, so there is an urgent need for a new monitoring system with high concurrent processing capabilities. [0003] As we all know, the value of data decreases rapidly with the passage of time, and the time requirement for monitoring big data is more urgent. If the system takes a long time to process the fault data after monitoring,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/30G06F11/32
Inventor 刘桂海陈忠强黄伟鞠强魏亮周国庆
Owner QINGDAO TELD NEW ENERGY TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap