Distributed internet data acquisition system and method based on event-driven model

Abstract

The invention discloses a distributed internet data acquisition system and method based on an event-driven model, and relates to the technical field of network search. The distributed internet data acquisition system comprises a console module, a data acquisition engine module, a data storage module and a log service module, and the distributed internet data acquisition system runs on a containerarrangement engine, wherein the console module configures data acquisition and data storage; the data acquisition engine module completes data acquisition according to configuration of the console module; the data storage module is connected with the data acquisition engine module and completes data storage according to configuration of the console module; the console module, the data acquisitionengine module, the data storage module and the log service module respectively comprise one or more services; and the services are mutually decoupled and deployed on the container orchestration enginein an independent Docker mirror image form. The distributed internet data acquisition system has the capability of quickly and dynamically expanding and shrinking the capacity, supports the requirement of daily acquisition of TB quantity-level mass data, and supports data acquisition of thousands of websites with different sources at the same time.

Description

technical field [0001] The invention relates to the technical field of network search, in particular to a distributed Internet data collection system and method based on an event-driven model. Background technique [0002] The rapid development of modern information technology has resulted in explosive growth of data and information contained on the Internet. In recent years, the proposal and application of big data has made people further realize the important value of Internet data. Internet data is increasingly regarded as digital oil, which can provide the underlying information driving force for governments, financial institutions, banks, traditional enterprises and other institutions. Therefore, for information scattered everywhere on the Internet, professional Internet data collection technology (also known as web crawler) is needed to collect these massive data in a timely and large-scale manner. [0003] The types of data on the Internet are abundant, and the pres...

AI Technical Summary

Problems solved by technology

[0005] 1. The train collector is client software, so it is not suitable for distributed deployment;

[0006] 2. The performance of the train collector is limited by the performance of the physical machine where the client is located;

[0007] 3. Combining the above points 1 and 2, the train collector system cannot meet the needs of real-time large-scale data collection

[0009] 1. Although it is a distributed crawler system, when the system needs to be expanded, the Archer collection system needs to be specially arranged and configured on the newly added node machine. Therefore, the cost of system expansion is high and cumbersome;

[0010] 2. There is no rapid shrinkage mechanism;

[0011] 3. Each machine node is relatively independent, and the operation and maintenance pressure is high;

[0012] 4. Does not support visual user interface, and has high technical requirements for users

[0015] 1. It is only suitable for writing small-scale crawler projects. When facing large-scale crawling tasks, it is necessary to adjust the code logic from the bottom of the system;

[0016] 2. It mainly provides code execution management and lacks support for distributed scheduling;

[0017] 3. Does not support visual user interface, and has high technical requirements for users

Images