Dynamic webpage crawler method and system based on substitute work mode

Abstract

The invention discloses a dynamic webpage crawler method and system based on a work substitution mode. The method comprises the steps of receiving service information, configuring crawler parameters,performing service evaluation, and performing preparatory work; allocating system resources, and initiating service crawlers of a plurality of independent processes; crawling the original URL of the dynamic webpage by adopting a simulated browser mode, and returning the URL of the target static data content; reviewing the validity and non-repeatability of the URL, reviewing the crawling task afterreviewing, constructing a production task message list, and initiating a production crawler of a plurality of threads; crawling a static URL page by adopting an automatic program mode, and returningtarget data and an attachment file; processing and storing the returned content; deriving data. According to the method, the business crawler and the production crawler are constructed respectively, different crawling strategies are adopted for the dynamic webpage and the static content on the basis of the substitution mode, system resources are utilized to the maximum extent, and large-scale andrapid crawling of dynamic webpage data is achieved.

Description

technical field [0001] The invention relates to the technical fields of Internet information retrieval and search engines, in particular to a dynamic web page crawling method and system based on an OEM mode. Background technique [0002] Web crawlers are an important part of Internet search engines, and are mainly used to crawl data in web pages on the Internet and build indexes for search engines. Whether the amount of crawling is large enough determines whether the content of the search engine is rich and whether the crawling is immediate, directly affects the overall effect of the search engine. In the context of big data, web crawlers are also widely used to capture network data such as Internet public opinion, commodity transactions, cultural and sports entertainment, and provide massive basic data for further data mining and data analysis. [0003] The working principle of a general web crawler is to obtain the HTML data of the web page by accessing the URL of the tar...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is that the existing crawler method is inefficient, the task scheduling is simple, and cannot meet the needs of large-scale crawling tasks. The purpose is to provide a proxy-based A method and system for dynamic webpage crawling in industrial mode, solving the problem of simple and efficient large-scale completion of crawling tasks

Images