Data collection system and method

A data acquisition system and data acquisition technology, applied in the field of data processing, can solve the problems of difficult to deal with diversified enterprise data, multiple storage methods and concurrent storage methods, etc., to achieve the effects of achieving diversity, controlling load balance, and speeding up processing cycles.
CN104065741AActive Publication Date: 2014-09-24YONYOU NETWORK TECH
7 Cites 30 Cited by

Patent Information

Authority / Receiving Office
CN Β· China
Current Assignee / Owner
YONYOU NETWORK TECH
Publication Date
2014-09-24

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a data collection system based on a Zookeeper cluster. The system comprises a message queue establishing unit and the Zookeeper cluster. The message queue establishing unit is used for establishing different message queues according to different data sources. The Zookeeper cluster is used for monitoring the states of a plurality of collectors, and enabling any message datum in the message queues to be collected by only one collector, wherein each message queue corresponds to a persistence node of the Zookeeper cluster. Each collector obtains message data under the coordination of the Zookeeper cluster, and corresponds to the temporary node of the persistence node. Correspondingly, the invention further provides a data collection method based on the Zookeeper cluster. According to the technical scheme, different collection modules can be arranged according to different data sources, different storage strategies are set according to different storage requirements, and diversity of data collection and storage is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the technical field of data processing, in particular to a data acquisition system based on a Zookeeper cluster and a data acquisition method based on a Zookeeper cluster. Background technique

[0002] With the advent of the big data era, the amount of enterprise data is increasing dramatically, and the data generated by various enterprise applications needs to be collected and stored on the big data platform in a real-time, efficient and secure manner.

[0003] The current data collection system has a variety of solutions to choose from, such as distributed data collection tools Apache Flume, Facebook Scribe, etc. However, most of the current data acquisition systems only aim at a specific data source and store the specific data source in a specific file system or data warehouse, which makes it difficult to deal with the diverse enterprise data in practical applications and the following The concurrent storage method based on...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More