Method and system for crawler paging selection of big data network

Abstract

The invention discloses a method and system for crawler paging selection of a big data network. The method comprises the steps that a crawler script is analyzed; a matching character is obtained to match label information in crawler script contents; feature values of matched labels are stored into a URL queue; a URL connecting address in the URL queue is acquired, and verified; the verified URL connecting address is acquired for address matching; and a web page of the URL address with successful address matching is analyzed, and paging information is acquired. The system comprises a first analysis module, a first matching module, a storage module, an acquisition module, a second matching module, a second analysis module and a configuration module. According to the invention, the problem that web page data cannot be crawled cyclically when an HTML structure of a page button in the current web page is changed can be solved; paging labels of the data can be recognized accurately; interruption of the cyclic data crawling can be prevented effectively; and crawling efficiency of the web pager data is increased.

Description

Technical field [0001] The invention relates to the technical field of big data analysis and processing, in particular to a big data web crawler paging selection method and system. Background technique [0002] With the rapid development of the Internet, the World Wide Web is generating all kinds of data at all times. At present, the total number of websites in China is about 4.54 million, and the number of web pages exceeds 200 billion. The surge of data contains amazing value. How to effectively extract and use this information has become a huge challenge. How to make these complex and disorderly Internet data generate value, how to turn the World Wide Web into their own database, how to make enterprises easily control these massive data information to innovate and quickly insight into business opportunities, search engines, such as traditional general search Engines such as Google and Baidu, as a tool to assist people in retrieving information, have become the entrance and g...

AI Technical Summary

Problems solved by technology

When crawling data, due to changes in the HTML structure of the page buttons in some web pages, the corresponding label information cannot be located, resulting in the problem that web page data cannot be crawled cyclically. For example, the HTML structure of the "next page" in some web pages is in After clicking "Next Page" several times, it will change. After the data structure changes, the "Next Page" button cannot be located through the original HTML locator, and web page data cannot be crawled in a loop.

Images