Distributed crawler system

Abstract

The invention relates to a distributed crawler system which comprises a URL read-write module, a URL capture module, a document analysis module and a persistence module. The URL read-write module is used for reading the URL from the input stream based on the Map end of the MapReduce and writing the URL into the output stream; The URL capturing module is used for taking the URL written into the output stream as an access address, and downloading a target document corresponding to the access address based on the Map terminal according to a preset network access mode; The document analysis moduleis used for extracting target data in the target document according to a preset mode based on the Map end; And the persistence module is used for storing the target data into the Hadoop distributed file system according to a preset path and a persistence rule based on the Map terminal. According to the scheme, the distributed crawler system is modularized, information interaction is carried out by transmitting data among the modules, the expandability, the usability and the maintainability of the system are improved, the scheduling capability of the system is improved, and the crawling performance of the system can be brought into play.

Description

technical field [0001] The invention relates to the technical field of web crawlers, in particular to a distributed crawler system. Background technique [0002] The advent of the Internet era has brought about a rapid expansion of the amount of information, and big data and cloud computing have also emerged as the times require. Internet companies, large communication companies, and sales companies generate huge amounts of logs and user behavior information every day. The characteristics of big data, such as huge data volume, complex data types, low value density, and fast processing speed, make traditional centralized web crawlers limited by web page coverage and crawling time performance bottlenecks, and insufficient system scheduling capabilities, resulting in system Crawling performance is poor. Contents of the invention [0003] In view of this, the purpose of the present invention is to provide a distributed crawler system to overcome the limitations of the current...

AI Technical Summary

Problems solved by technology

[0003] In view of this, the purpose of the present invention is to provide a distributed crawler system to overcome the limitations of the current traditional centralized web crawler due to the bottleneck of Web page coverage and crawling time performance, insufficient system scheduling capabilities, and poor system crawling performance. bad question

Images