Studying method and system based on increment Q-Learning

Abstract

The invention relates to a network page creeping method and a creeping system. In the method, the system recalculates the Q value of each knot function on the hyperlink link corresponding to a new creeping web page; the system re-disperses the function Q value according to the newly calculated function Q value to form a new sample; then an NB classifying device is retrained to obtain a new Q value classification model which is utilized for recalculating the Q value of each candidate URL in a URL queue; finally the IQ-Learning arithmetic leads a page correlation evaluating device to carry out increment learning. The innovation point of the system structure of the invention lies in the addition of the Q-Learning on-line sample generator which carries out analysis and evaluation to the pages obtained by on-line creeping and generates new positive-example samples or negative-example samples so as to cause the increment leaning to be possible. The technique introduced by the invention effectively enhances the obtaining rate of theme crawlers.

Description

technical field [0001] The invention relates to a learning method and system based on incremental Q-Learning, which is an incremental Q-Learning learning method and system for quickly and effectively retrieving information required by users from the World Wide Web. Background technique [0002] Web crawler (Web Crawler or Spider, Robot) is an information collection system. It collects Web pages by downloading them and traversing the Web along the hyperlinks in the crawled pages. General web crawlers are usually used in general search engines as a page collection system for search engines. It usually traverses the Web in a breadth-first mode (that is, non-selective), and strives to collect as many Web pages as possible within a limited crawl cycle. [0003] Web crawlers use a specific crawling strategy to periodically collect as many web pages as possible, and then submit them to the automatic indexing system; the indexing system builds an index library based on the corresp...

AI Technical Summary

Problems solved by technology

[0011] In order to solve the problem existing in the existing theme crawling technology, for relatively "narrow" target topics, the crawling harvest rate of the system is low, and the generated page (hypertext) classification model and hyperlink evaluation model are static. cannot be updated, and lacks the ability to learn incrementally online, resulting in the initial sample pages (including pages in the topic hierarchy directory and sample pages provided by users) becoming the main factor determining the performance of hypertext classifiers and hyperlink evaluators

[0012] In order to solve the problem existing in the existing theme crawling technology, for relatively "narrow" target topics, the crawling harvest rate of the system is low, and the generated page (hypertext) classification model and hyperlink evaluation model are static. cannot be updated, and lacks the ability to learn incrementally online, resulting in the initial sample pages (including pages in the topic hierarchy directory and sample pages provided by users) becoming the main factor determining the performance of hypertext classifiers and hyperlink evaluators

Images