Method for incrementally capturing webpage contents

Abstract

The invention discloses a method for incrementally capturing webpage contents. The method comprises the steps of establishing a webpage content basic resource library in a global search traversal manner, next, extracting URLs of webpage contents in a changing update status, adding the URLs and corresponding webpage analysis rules to a distributed task queue, downloading the webpages and creating webpage DOM objects, extracting webpage content metadata according to the analysis rules, and replacing the updated information after comparing the extracted metadata with the webpage contents in the original basic library, or inserting the webpage content information to URLs completely nonexistent in the basic resource library. The method for incrementally capturing the webpage contents does not need to traverse the webpage contents of all the URLs while checking update of the whole resource library, and only needs to traverse the webpage contents under the URLs having update change identifiers, and therefore, incremental capture on the webpage contents is realized, the capture workload is reduced and the update efficiency is improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to an incremental webpage content grabbing method. Background technique [0002] At present, crawler technology is widely used in crawling webpage content. Massive webpage information is usually obtained through a greedy and carpet-like web crawler method and a corresponding resource library is established. However, in the era of rapid Internet development, the update speed of webpage information is also extremely fast. In this case, due to the requirements for timeliness, freshness and other indicators, the value of the existing webpage content information resource library is reduced, which cannot meet the corresponding requirements. need. The existing incremental crawling method usually traverses all web pages and extracts web page information with higher freshness compared with existing resources. This method is very inefficient and has relatively high requirements for time, s...

AI Technical Summary

Problems solved by technology

However, in the era of rapid Internet development, the update speed of webpage information is also extremely fast. In this case, due to the requirements for timeliness, freshness and other indicators, the value of the existing webpage content information resource library is reduced, which cannot meet the corresponding requirements. need

The existing incremental crawling method usually traverses all web pages and extracts web page information with higher freshness compared with existing resources. This method is very inefficient and has relatively high requirements for time, space and network bandwidth. High, in this case, an efficient method of incremental web page content crawling is particularly important

Images