Data acquisition monitoring method and device

A data acquisition and data technology, applied in the direction of data exchange network, digital transmission system, electrical components, etc., can solve the problem of a large number of acquisition tasks, business log data failures that cannot be quickly found and accurately located, and the Flume system lacks data acquisition and monitoring technology, etc. question

Active Publication Date: 2017-06-30
BEIJING SOHU NEW MEDIA INFORMATION TECH
View PDF11 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, in the context of big data, the business log data that needs to be collected is distributed on many servers, and a single server has multiple business log data that need to be collected separately, resulting in a large number of collection tasks
In the process of di

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition monitoring method and device
  • Data acquisition monitoring method and device
  • Data acquisition monitoring method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0056] The data acquisition and monitoring method in the embodiment of the present invention is applied to a Kafka-based distributed data acquisition system that can be monitored. HDFS or other reliable media for persistent storage of data. In this system, Slave performs the actual collection task. Slave is divided into two types according to different collection functions. Among them, Source (the first type of slave server) is responsible for collecting data ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data acquisition monitoring method and device. The heartbeat data reported by a first type of slave server are acquired, wherein the heartbeat data include data volume information of files required to be acquired by the first type of slave server and the completely acquired data volume information of the files; and whether the first type of slave server fails to work for the acquisition process of the files is judged according to the data volume information of the files required to be acquired by the first type of slave server and the completely acquired data volume information of the files, and the data after the position of the failure in the files is the last completely acquired data of the files are located when the first type of slave server fails to work for the acquisition process of the files. The failure occurred in the distributed acquisition process of the business log data can be rapidly discovered and accurately located based on the heartbeat data.

Description

technical field [0001] The invention relates to the technical field of data acquisition, in particular to a data acquisition and monitoring method and device. Background technique [0002] With the development of the Internet, the business volume has increased rapidly, the scale of server clusters providing services has become larger and larger, and the types of services provided have become more and more complex. The reliability of business log data collection under the background of such big data is becoming more and more important. [0003] The current mainstream business log data collection system is a highly available, highly reliable, and distributed Flume system for massive log collection, aggregation, and transmission provided by Cloudera. At present, the business log data collection process based on the Flume system is as follows: First, the Flume system is used to collect business log data in a distributed manner, then the collected business log data is aggregated...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L12/24H04L12/26
CPCH04L41/0631H04L41/0677H04L41/069H04L43/04H04L43/10
Inventor 赵剑雄邵位潘子凡
Owner BEIJING SOHU NEW MEDIA INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap