Dicycle crawler system based on Spark Streaming and running method thereof

Abstract

The invention discloses a dicycle crawler system based on Spark Streaming and a running method thereof. According to the dicycle crawler system, a page downloading module is responsible for downloading pages, DNS information cached in a DNS cache module is called during downloading, and DNS parsing is accelerated to shorten time of page downloading; a URL distribution scheduling module is responsible for calling a URL extraction module, URLs are extracted from newly downloaded pages, a URL repetition removing module is called to filter repetitive URLs, and the URLs are distributed to the pagedownloading module; a page scheduling module is responsible for calling a page analyzing module, it is determined that a current page accords with an extraction model, data extraction tasks are distributed to a page extracting module, the page extracting module is used for extracting data from the current page, and the extracted data is saved in a distributed storage system; parameter configuration and monitoring management of a whole crawler are both controlled in a web background. The dicycle crawler system based on Spark Streaming and the running method thereof solve the horizontal extension problem of traditional crawlers and have the advantages that module division is clear, a running mechanism is stable and efficient, crawler rule configuration is concise and the universality is high.

Description

technical field [0001] The invention relates to a dual-period crawler system based on Spark Streaming and an operating method thereof, belonging to the technical field of distributed crawlers. Background technique [0002] The crawler system is a system for collecting massive and scattered Internet data, and it is the basis of the search engine system. Big data has developed rapidly in recent years and is hot, not only because of the large capacity of data, but also because of the emphasis on the analysis of full sample data. Internet data contains a lot of valuable information and is an important data source for big data. Its organizational forms are also flexible and diverse. Most websites adopt anti-crawler strategies such as dynamic loading, which brings great challenges to the information collection and storage of traditional crawlers. challenge. Traditional crawlers run on a single node, which has poor horizontal scalability. The number of crawler threads and data st...

AI Technical Summary

Problems solved by technology

[0004] Purpose of the invention: Aiming at the problems existing in the reptile system in the prior art, the present invention provides a dual-cycle crawler system and its operation method based on SparkStreaming, which solves the problem of horizontal expansion of traditional crawlers, and has clear module division, stable and efficient operation mechanism , Crawler rule configuration is simple, versatile, etc.

Images