Self-adaptive incremental deep web data source discovery method

Abstract

The invention discloses a self-adaptive incremental deep web data source discovery method. According to the method, the deep web data source discovery processes comprise a website positioning stage and an in-web searching stage, and in the website positioning stage, a website discovery mechanism is introduced so that website data can be efficiently expanded and the creep effect can be improved; a self-adaptive sorting mechanism is adopted in website and in-web linkage so that a deep web website and a queryable form can be discovered more rapidly. The method achieves automatic incremental efficient deep web data source acquisition, can be applied to deep web data integration and a hidden web crawler, and meanwhile is also suitable for building on-line database catalog websites.

Description

technical field [0001] The invention belongs to information retrieval and data mining in the field of computers, and in particular relates to an adaptive incremental deep network data source discovery method, which can automatically and efficiently discover deep network data sources according to fields. Background technique [0002] With the explosive growth of Internet data, more and more websites adopt network database technology. A large number of pages on the Internet are dynamically generated by the database. This information cannot be crawled through static links, but must be obtained by submitting queries. Since search engine crawlers do not have the ability to automatically fill in forms, this part of the data cannot be indexed by search engines and is hidden behind the network database. This part of the data is called the deep web or dark web. [0003] The White Paper on Deep Network released by BrightPlanet in 2001 made a relatively comprehensive macro-statistics o...

AI Technical Summary

Problems solved by technology

[0004] Existing data source discovery mechanisms can be mainly divided into two categories: general crawlers and theme crawlers. General crawlers generally crawl in an exhaustive manner, which will download a large number of irrelevant pages, resulting in low efficiency; theme crawlers crawl according to domains. Use the page classifier to filter out the pages irrelevant to the topic, and then use the link classifier to filter the links to speed up the crawling process, but due to factors such as topic drift and link classifier precision, the efficiency of crawling will be reduced; in addition, due to the sparse distribution of data sources, the topic The crawler speeds up the crawling process by setting termination conditions, which will result in a large number of pages without the opportunity to visit, and the page classifier and link classifier will also filter out many pages and links, so it is difficult to ensure high coverage

Images