Distributed internet data acquisition system and method based on event-driven model

Abstract

The invention discloses a distributed Internet data acquisition system and method based on an event-driven model, and relates to the technical field of network search. On the engine; the console module configures data collection and data storage; the data collection engine module completes data collection according to the configuration of the console module; the data storage module is connected with the data collection engine module, and completes data storage according to the configuration of the console module; The console module, data collection engine module, data storage module, and log service module each contain one or more services; the services are decoupled from each other and deployed on the container orchestration engine in the form of an independent Docker image. The present invention has the capability of rapidly and dynamically expanding and shrinking capacity, supports the requirement of daily collection of terabyte-level massive data, and supports data collection of thousands of websites from different sources at the same time.

Description

technical field [0001] The invention relates to the technical field of network search, in particular to a distributed Internet data collection system and method based on an event-driven model. Background technique [0002] The rapid development of modern information technology has resulted in explosive growth of data and information contained on the Internet. In recent years, the proposal and application of big data has made people further realize the important value of Internet data. Internet data is increasingly regarded as digital oil, which can provide the underlying information driving force for governments, financial institutions, banks, traditional enterprises and other institutions. Therefore, for information scattered everywhere on the Internet, professional Internet data collection technology (also known as web crawler) is needed to collect these massive data in a timely and large-scale manner. [0003] The types of data on the Internet are abundant, and the pres...

AI Technical Summary

Problems solved by technology

[0005] 1. The train collector is client software, so it is not suitable for distributed deployment;

[0006] 2. The performance of the train collector is limited by the performance of the physical machine where the client is located;

[0007] 3. Combining the above points 1 and 2, the train collector system cannot meet the needs of real-time large-scale data collection

[0009] 1. Although it is a distributed crawler system, when the system needs to be expanded, the Archer collection system needs to be specially arranged and configured on the newly added node machine. Therefore, the cost of system expansion is high and cumbersome;

[0010] 2. There is no rapid shrinkage mechanism;

[0011] 3. Each machine node is relatively independent, and the operation and maintenance pressure is high;

[0012] 4. Does not support visual user interface, and has high technical requirements for users

[0015] 1. It is only suitable for writing small-scale crawler projects. When facing large-scale crawling tasks, it is necessary to adjust the code logic from the bottom of the system;

[0016] 2. It mainly provides code execution management and lacks support for distributed scheduling;

[0017] 3. Does not support visual user interface, and has high technical requirements for users

Images