Theme webpage crawling method and theme crawler system

Abstract

The invention provides a theme webpage crawling method and a theme crawler system. The method comprises the steps of obtaining un-crawled links from a first to-be-crawled link set comprising seed links; determining first correlation degree and second correlation degree corresponding to target webpages corresponding to the obtained links, wherein the first correlation degree and the second correlation are the correlation degree between target text content in the target webpages and an appointed theme and between target links and the appointed theme; determining temperature values of the targetwebpages according to the first correlation degree and the second correlation, and storing to-be-displayed content of the target webpages; placing the target links into a second to-be-crawled link setif the temperature values of the target webpages are greater than or equal to a preset temperature value; and obtaining the un-crawled links of which correlation degree with the appointed theme is the highest, from the second to-be-crawled link set and crawling the obtained un-crawled links if un-obtained links do not exist in the first to-be-crawled link set. According to the method and the system, a user can obtain a great number of webpages correlated to the appointed theme from a network.

Description

technical field [0001] The present invention relates to the technical field of webpage crawling, in particular to a method for crawling theme webpages and a theme crawler system. Background technique [0002] With the rapid development of the Internet, people have ushered in an era of information explosion, and all kinds of information are flooding all aspects of life. In order to facilitate the acquisition of information, search engines have emerged. People can quickly retrieve information on many web pages through search engines, and search engines have improved the efficiency of people's access to information. At present, commonly used search engines such as Google, Baidu, etc. are all general search engines. This type of search engine tries to obtain all the resources on the Internet. However, people's needs are various. The content of webpages with specified topics can be obtained, but general search engines cannot meet the personalized needs of users. Contents of th...

AI Technical Summary

Problems solved by technology

At present, the commonly used search engines such as Google and Baidu are all general search engines. This type of search engine tries to obtain all the resources on the Internet. However, people's needs are diverse. Get the webpage content with the specified topic, but general search engines cannot meet the personalized needs of users

Images