Quick collection system and method for distributed internet data

Abstract

The invention discloses a quick collection system for distributed internet data. The system comprises five layers including a seed website setting node, a hyperlink collection layer, a real-time queue, a webpage downloading and parsing layer and a webpage data storage layer, wherein the seed website setting node is used for setting each parameter and each extracting rule for storing a data source; the hyperlink collection layer is used for requesting the hyperlink list webpage of the data source and extracting the hyperlink of a target webpage; the real-time queue is used for accessing a URL (Uniform Resource Locator) hyperlink extracted by the hyperlink collection layer, the extraction rule corresponding to the URL hyperlink and the accessed URL hyperlink; the webpage downloading and parsing layer is used for requesting and parsing the URL hyperlink which is not accessed in the real-time queue and carrying out formatting extraction on specific data; and the webpage data storage layer is used for storing target data obtained by the formatting extraction carried out by the webpage downloading and parsing layer. By use of the system, data collection is carried out by a distributed layered cooperation way, and the system application requirements including high data collection quantity, more data sources and high instantaneity requirements can be coped with.

Description

technical field [0001] The invention belongs to the technical field of Internet big data collection, and in particular relates to a distributed Internet data fast collection system and collection method. Background technique [0002] The rapid development of the Internet has brought society into the information age with highly developed and open data, and the era of big data has come. Data plays an extremely important role in business operations, government decision-making, and social dynamic analysis, and how to collect data on a large scale and quickly has become a technical focus. However, from the perspective of existing technical solutions, data collection methods need to be improved. Traditional Internet data collection mainly uses web crawlers as the main tool, and takes structured or semi-structured text data as the object of data collection. A web crawler is a program or script that automatically walks and crawls Internet text web pages according to certain rules. ...

AI Technical Summary

Problems solved by technology

[0003] In view of this, the present invention provides a distributed Internet data fast collection system and collection method for the problems of large amount of data collection, many data sources and low real-time performance

Images