Webpage classification recognition method based on comprehensive subject term vertical search and focused crawler

Abstract

The invention discloses a webpage classification recognition method based on comprehensive subject term vertical search and focused crawler, and belongs to the technical field of webpage search engines. According to the method, research is performed aiming at a webpage classification recognition method in a subject term vertical search engine which is dynamically changed in a webpage, and the judgment of a fact that whether a dynamically changed webpage is related to a subject term is mainly searched; by computing the subject term correlation degree in the webpage, a URL highly related to a comprehensive subject term is screened out and enters a queue for crawl; classified information of the webpage is obtained through vertical search and focused crawler technologies; a webpage classification recognition model and algorithm are designed; different classifications of URLs are obtained through the recognition of the dynamically changed webpage; accurate webpage search is provided for users, and the webpage classification of an unknown URL can be further provided. The method has very wide significance and a high application value for the classification recognition of the dynamic webpage.

Description

technical field [0001] The invention relates to the technical field of webpage search engines, in particular to a method for classifying and identifying webpages based on vertical search of comprehensive subject words and focused crawlers. Background technique [0002] With the increasing popularity of vertical search engines, as a key technology of vertical search engines—focused crawlers are becoming more and more important. Focused crawler is a program that automatically downloads web pages. It selectively visits web pages and related links on the World Wide Web according to the established crawling target to obtain the required information. The main processing object of the crawler is URL. Get the required file content, and then do further processing on it. [0003] With the rapid growth of the Internet, the amount of information on the Internet is also presenting explosively. People pay special attention to how to obtain effective information from the massive amount of...

AI Technical Summary

Problems solved by technology

[0016] This method has certain difficulties for the following reasons: it is difficult for the focused crawler to select the crawling queue that is closely related to the topic information from the URL queue to be crawled; in the process of URL extraction, the web crawler uses search strategies such as depth and breadth, which is easy to generate The "curse of dimensionality" problem; many existing open source crawler systems are weak in obtaining structured information from crawled webpages; existing focused crawler strategies are difficult to adapt to the dynamic changes in the content and structure of webpages

To sum up, the recognition rate of different categories of webpages in the traditional focused crawler technology is low, and another method must be found

Images