An efficient data collection method and system based on deep web crawler

Abstract

The invention discloses a high efficiency data acquisition method and system based on deep web reptiles; the method comprises the following steps: using a known name as the keyword so as to carry out preliminary search sampling, thus obtaining corresponding number information and number rules; grouping all number information according to number rules, and ranking the number information in an ascending order, wherein a data gap is built between every two adjacent number information; traversing and searching the data according to the number information and data interval in a rising sequence, thus obtaining the data complete set. The method and system can grab various industry information data including but not limited to: enterprise business information, book information, goods information, and trial files; the method and system can grab deep web data on industrial websites, can accurately obtain related data complete set, and can finish mass effective data acquisition in a short time.

Description

technical field [0001] The present invention relates to the technical field of network data collection, in particular to an efficient data collection method and system based on deep web crawlers, which can be applied to capture data such as enterprise business information, book information, commodity information, and trial documents. Background technique [0002] The concept of the deep web is defined relative to the surface web, referring to content that cannot be obtained by ordinary search engines. The amount of data hidden in the deep web is huge, but traditional search engines ignore these high-quality data hidden behind search forms. Therefore, it is imperative to design an engine that can obtain deep network data, and the use of these high-quality data can bring unexpected effects. [0003] However, because the data in the deep web is hidden behind various search interfaces and cannot be accessed directly through hyperlinks, they must be viewed by using some keyword ...

AI Technical Summary

Problems solved by technology

[0005] One is to use relevant names as keywords to obtain the complete set of information. The disadvantage of this method is: because the names of all the information to be collected are not necessarily known, it is basically impossible to obtain the complete set, and the processing efficiency is low;

[0006] One is to use the relevant serial number information to traverse and search one by one to obtain the complete set of information. The disadvantage of this method is that the collection output is unstable due to the discontinuous characteristics of the serial number information (such as: reserved space, information discarded, not yet started, etc.) , not efficient, unable to achieve the purpose of efficient collection

Images