Method and system for parallel square crossing network data collection

Abstract

The invention provides a method and system for parallel square crossing network data collection. The method comprises the steps that S1 a collection method is determined, and S2 tasks are updated at regular time, wherein if a common URL list collection method is determined, data and metadata of a webpage to be downloaded are directly downloaded according to a URL list, and if a square crossing collection method is determined, a webpage to be downloaded is searched by utilizing a crossing keyword list, and data and metadata of the webpage is downloaded. Scanning inspection is carried out on all webpages from the first layer to the current layer in a downloaded webpage, and when the last modification time of a newly downloaded webpage is later than the last modification time of a downloaded webpage, data collection is carried out on the newly downloaded webpage, and webpage data records are updated. By means of the method and system for parallel square crossing network data collection, potential risks and cost in the multithreading technology are avoided, potential risks and cost in multithreading parallel collection are lowered, parallel collection of mass data can be carried out more stably and more efficiently, and data reading and inquiring efficiency is improved.

Description

technical field [0001] The invention relates to the field of the Internet, in particular to a parallel vertical cross network data collection method and system. Background technique [0002] A web crawler is a program that automatically extracts web pages. It downloads web pages from the World Wide Web for search engines and is an important component of search engines. Traditional crawlers start from the URL of one or several initial webpages, obtain the URLs on the initial webpage, and continuously extract new URLs from the current page and put them into the queue during the process of crawling webpages until a certain stop condition of the system is met. In addition, all the webpages crawled by the crawler will be stored by the system, analyzed, filtered, and indexed for later query and retrieval. [0003] A combination of four strategies is usually considered when designing a web crawler: [0004] Selection strategy: decide which pages to download; [0005] Revisit pol...

AI Technical Summary

Problems solved by technology

[0019] Poor quality crawlers that may crash servers or routers, or try to download pages they cannot handle

[0020] Personal crawler, if too many people use it, it may be that the network or server is blocked

[0021] The disadvantage of using objects as the collection target is that the hierarchical structure of the collected web pages is too single, which is not convenient for subsequent data analysis and processing; the defect of using multi-threads as the collection method is mainly that when the number of threads is too large, the scheduling , resource occupation costs, etc. will increase accordingly, and there is a high risk of thread deadlock

Images