Vertical intelligent crawler data collecting method based on webpage data capture

Abstract

The invention relates to a vertical intelligent crawler data collecting method based on webpage data capture. The vertical intelligent crawler data collecting method is characterized by comprising the steps that firstly, a crawler data initial inlet address is configured to a starting module through starting / stopping of an inlet configuration module; then a crawler control system performs a depth-first algorithm to perform traversal webpage capture according to set capture rules and a set capture process; finally, crawlers perform analysis extraction on webpage data and storing extracted two-dimensional structure data through a rule sequence pair of a rule configuration system. Therefore, the universality requirements of the crawlers can be met, analysis rule configuration, webpage depth and thread capturing, database configuration or index configuration can be increased on specific service logic, and intelligent information capture can be started. An intelligent crawler frame can be effectively formed, meanwhile automatic data archival and classified storage can be achieved, and a mode that distributed key values are adopted for database storage can be adopted.

Description

technical field [0001] The invention relates to a data collection method, in particular to a vertical intelligent crawler data collection method based on webpage data capture. Background technique [0002] Crawler, also known as spider, is not the name of an insect, but a computer program that people use to continuously extract links to web pages through customized entry URLs on the Internet, and then crawl and extract deeper unknown links based on these links. Going forward, such program crawling is described as a crawler-like action, which is called a crawler. A crawler is a program that automatically obtains web content and is an important part of a search engine. [0003] Regarding vertical crawlers, the core technology of vertical search is actually the technology of intelligent crawlers, which is how to capture directional or non-directional web pages and analyze them to obtain formatted data. It is mainly used to accurately extract regular two-dimensional table data,...

AI Technical Summary

Problems solved by technology

[0004] However, in the past, the traditional vertical grabbing program needed to strongly couple the parsing and grabbing logic to the entire module, which was inconvenient for later maintenance and function expansion, high maintenance cost, low efficiency, and no framework scalability

Images