Universal distributed acquisition system

Abstract

The invention discloses a universal distributed acquisition system which comprises a seed storehouse, a task scheduling module, a data capture module and a regular text page storehouse, wherein the seed storehouse is used for storing a URL (Uniform Resource Locator) of a required site and setting an information source category and an acquisition time interval; the task scheduling module is used for coordinating a task load of each acquisition node; the data capture module is used for performing information capture on the distributed acquisition tasks, wherein the information capture is divided into list page capture and regular text page capture; the task scheduling module and the data capture module comprise servers and clients and both adopt distributed communication frameworks; and the regular text page storehouse is used for storing the parsed regular text web page link and providing a site gate for the regular text page in the data capture module.

Description

technical field [0001] The present invention relates to a distributed high-concurrency collection system for Internet-wide data, in particular to an efficient and real-time data collection system in a big data environment, and in particular to a general distributed collection system. Background technique [0002] In recent years, with the rapid development and popularization of computer and information technology, the scale of industrial application systems has expanded rapidly, and the data generated by industrial applications has grown explosively. The current total data volume of Baidu has exceeded 1000PB, and the webpage data that needs to be processed every day reaches 10PB~100PB; the cumulative transaction data volume of Taobao is as high as 100PB; Twitter publishes more than 200 million messages every day, and Sina Weibo posts 80 million posts every day; China Mobile's telephone communication record data in a province can reach 0.5PB to 1PB per month; the road vehicle...

AI Technical Summary

Problems solved by technology

For the collection of a large amount of data in the current network environment, the traditional manual data acquisition and single-node crawler acquisition of data can no longer meet the needs. At present, there are some distributed collection systems, but there are certain problems in each process of data collection. Bottleneck, it is difficult to efficiently and rationally utilize the performance of the machine and expand on a large scale

Images