Information crawling method and device

Abstract

The disclosure provides an information crawling method and device. The method comprises: acquiring a configuration file generated according to webpage structure and information crawling need of a target website; executing an information crawling task for the website according to the configuration file. The information crawling method provided herein enables information crawling efficiency to be improved.

Description

Technical field [0001] The present disclosure relates to the field of Internet technology, and in particular, to an information crawling method and device for crawling information by configuring a crawling template. Background technique [0002] With the advent of the big data era, the importance of data has become increasingly prominent, and the collection of large amounts of data has become more and more important. At present, the methods of data collection are mainly divided into using internal data for collection and using the Internet for collection. A common technique is to encode the data that needs to be collected to capture the specified data. When using the Internet for data collection, it can be divided into using search engines for collection and using crawlers for crawling. [0003] The search engine platform can collect open information on the Internet across the entire network, with a wide range of collected data, fast update frequency, and large amount of collected...

AI Technical Summary

Problems solved by technology

[0003] The search engine platform can collect open information on the Internet across the entire network, with a wide range of collected data, fast update frequency, and a large amount of collected data. However, most of the collected information is fuzzy information, and the accuracy of the results is not high. And clean data, so when it is necessary to obtain accurate data, usually collect accurate data and usually use crawlers to accurately crawl information

[0004] Accurate crawling of information using crawlers usually requires custom coding for each type of target page, so it has the advantages of flexible data collection, accurate crawling information, and controllable crawling directions, but requires a large workload, and code maintenance is complex, difficult and Crawling speed is limited by the performance of a single machine

In addition, because the existing methods usually need to load the content of the page and download all the content on the page before crawling, it will take up a lot of resources when multiple web pages need to be crawled, resulting in a slow crawling speed

Images