Theme web portal crawler method

Abstract

The invention relates to the technical field of network information capture, in particular to a topic portal website crawler method. The method comprises the following steps of: analyzing and extracting a webpage page link, and designing a regular expression according to a theme website so as to identify a parent page link and a child page link; webpage content extraction: extracting the text content under the sub-page link, and storing the extracted text content in a static class; the data persistence storage being used for storing the text content extracted from each sub-page link; and incremental capturing: capturing the updated content in the theme webpage, re-extracting the link of the home page of the theme webpage during each incremental updating, and only processing the new link. The page obtained through the crawler program is almost not repeated, the required theme can be accurately obtained, the webpage containing the same content can be effectively prevented from being downloaded for multiple times, a large number of cpu resources are prevented from being wasted, and the load brought by database access is relieved.

Description

technical field [0001] The invention relates to the technical field of network information grabbing, in particular to a theme portal website crawling method. Background technique [0002] In the open environment of the Internet, the explosive growth of shared network information has provided people with a large number of information resources. However, this has also brought huge challenges. There are many types of information. more and more difficult. At this time, the search engine began to be born. Searching network information through keywords greatly facilitates people to search for information effectively and can meet most of the information needs. However, most search engines are based on horizontal search, and the main disadvantage of this method is that the returned search results have low accuracy and contain a lot of interference information. With the development of information diversification, this search strategy can no longer meet the specific needs of users. ...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problems that the above-mentioned current web theme crawler system captures webpages containing the same content once and downloads them multiple times, which wastes a lot of cpu resources and increases the load on database access, the present invention aims at content capture and The deduplication of the two links of incremental update proposes an efficient deduplication strategy, which is superior to traditional methods in terms of performance and scalability

Images