Deep web self-adapting crawling method based on minimum searchable mode

Abstract

The invention discloses a Deep Web self-adapting crawling method based on a minimum enquiry pattern. Aiming at the problem of the existing Deep Web crawling method that the crawling efficiency is low due to data isolated island, the invention firstly provides a conception of a minimum enquire pattern MEP and then provides an MEP generating algorithm and the self-adapting crawling method based on the MEP. The invention can cause an enquiry interface to be popularized to a minimum enquiry pattern set from a single textbox, a once enquiry is commonly determined by one MEP and keyword vector matched with the MEP, and a next enquiry with optimal expectation can be produced by a self-adapting way until enquiry stop conditions are satisfied. By using the minimum enquiry pattern, not only the form filling accuracy ratio is improved, but also the characteristics of all patterns can be fully utilized to select keywords so as to overcome the data isolated island problem better.

Description

technical field [0001] The invention belongs to the field of computer applications, and mainly relates to Web mining and information acquisition, in particular to a Deep Web self-adaptive crawling method based on the minimum queryable mode, which mainly solves the data island problem existing in the current crawling methods of the same kind. Background technique [0002] The goal of Deep Web crawling is to obtain as many Deep Web data records as possible [2], and the key lies in how to generate suitable crawling. At present, Deep Web crawling can be divided into two types of query methods based on prior knowledge and without prior knowledge. [0003] The crawling method based on prior knowledge needs to establish a corresponding prior knowledge base before crawling, and then generate queries under the guidance of prior knowledge. The shortcomings of this type of method are: 1. The query form is required to contain more information for learning prior knowledge. For query for...

AI Technical Summary

Problems solved by technology

The shortcomings of this type of method are: 1. The query form is required to contain more information for learning prior knowledge. For query forms that contain less information, such as a single text box form that only accepts keyword queries, its processing The ability is relatively limited; 2. One query needs to complete the filling of the entire form, which reduces the crawling efficiency

Compared with the crawling method based on prior knowledge, the crawling method without prior knowledge improves the crawling processing ability. However, this type of method still has the following two problems: 1. It can only crawl a single text box and the default The obtained keywords match the text box; 2. The selection of keywords for initial crawling lacks sufficient basis

[0020] After analysis and comparison, the Deep Web crawling method introduced in domestic and foreign literature does not involve the minimum queryable pattern extraction method of the Deep Web query form and the crawling method based on the minimum queryable pattern

Images