Webpage classification technology based on vertical search and focused crawler

Abstract

The invention provides a method for identifying webpage classification based on vertical search and focused crawler. The method comprises two parts, namely a webpage source code acquisition method and a webpage content analysis method, wherein the webpage content analysis method is a key method, and comprises two main parts, namely extraction of structured information of the webpage and crawling strategy of the focused crawler. First, a URL is selected from a navigation site URL list to acquire a source file of the URL; and then, all classified URL of the navigation URL sites can be identified and acquired by the webpage content analysis method. The key method in the method is the webpage content analysis method, which is to first extract the webpage structured information, then carry out URL snatch by a directional breadth-first search strategy based on webpage content feature, and finally store the snatched URL and corresponding website classification in a list Category.

Description

technical field [0001] The present invention is aimed at the research of the webpage classification identification method in the vertical search engine of the fixed navigation website list. It mainly studies how to effectively obtain the classification information of the webpage based on the vertical search and focused crawler technology, and designs the identification model and algorithm of the webpage classification. It involves multiple fields such as vertical search, focused crawler, web data extraction, machine learning, data mining and natural language. Background technique [0002] With the continuous expansion of information, people are increasingly inseparable from search engines. Although general search engines such as Baidu and Google provide people with a lot of convenience, with the diversification of people's needs and the higher and higher requirements for the quality of search results, general search engines can no longer meet people's requirements in some sp...

AI Technical Summary

Problems solved by technology

[0005] First, it is difficult for a focused crawler to judge how to crawl the web pages that are most likely to contain topic-related information from the queue of URLs to be crawled

[0006] Second, many open source crawler systems do not have the function of directional extraction of web page structured information from crawled web pages

[0007] Third, the content and structure of the same webpage often change, and it is difficult for the crawler-focused revisit strategy to adapt to this change

[0008] It can be seen from the above that it is difficult to accurately identify different types of web pages using traditional open source focused crawler technology

Images