Sorting type hidden network database data acquisition method

Abstract

The invention discloses a sorting type hidden network database data acquisition method. By adopting the method, a high redundancy phenomenon caused by sorting type hidden network return limitation canbe avoided, and the better effect is obtained with the minimum cost. The method considers the influences of return limitation on sorting type hidden network return data, starts from document frequencies, and utilizes the low document frequency of a sample set to select crawling keywords, the degree of coverage of crawling data is increased, and repeatability is reduced. The method firstly selectswords corresponding to the low document frequency from the document sample set as alternative words to the crawling keywords, so that the coverage rate of returned results is improved; a document frequency estimation method is used for estimating document frequencies of all the alternative keywords in a target database, the alternative words of which the document frequencies are smaller than a limit value of the number of return as the crawling keywords, the number of target database documents is maximized, the repetition rate of crawling structures is reduced, and a lot of data in a hidden network can be acquired within the minimum cost.

Description

technical field [0001] The invention relates to the technical field of information retrieval, in particular to a method for acquiring sorted hidden network database data. Background technique [0002] In today's information age, networks and information exist in all areas of our production and life. The World Wide Web includes not only the surface web that has been indexed by standard search engines, but also some webs that are dynamically generated by online databases or file systems but not indexed by search engines, called deep webs or hidden webs. According to relevant data in 2001, the number of hidden web pages is 500 times that of the surface web (500 billion hidden web pages, about 1 billion web pages on the surface web), and, compared with the surface web, the hidden web database stores a large number of high-quality information. However, the hidden network database cannot be directly accessed, and can only be obtained by entering keywords through the search inter...

AI Technical Summary

Problems solved by technology

However, these algorithms have not completely solved the problem of crawling word selection in sorting hidden web crawling, and the research on this aspect is still relatively lacking.

Images