Data crawling method and device, computer equipment and storage medium

Abstract

The invention provides a data crawling method and device, computer equipment and a storage medium, and the method comprises the steps: reading and analyzing a crawler configuration file to obtain a uniform resource locator of a starting target page serving as a current page; obtaining target data of a previous page of the current page, and generating and sending an access request carrying the target data of the previous page according to the uniform resource locator of the current page; extracting target data of the current page from response data of the access request; judging whether a next page exists in the current page or not; if the next page exists, acquiring a uniform resource locator of the next page, taking the next page as the current page, and circulating the steps until the next page does not exist in the current page; and obtaining final target data according to the target data obtained by the last page. According to the method and the device, each target page can be automatically drilled down, the page data of each level can be extracted and analyzed, and meanwhile, the data can be automatically merged and integrated for output.

Description

Technical field [0001] The present application relates to the field of data processing, in particular, to a data climb method, device, computer device, and storage medium. Background technique [0002] Currently, climbing methods commonly used based on the SCRAPY framework are divided into two. [0003] One is based on inheriting the reptiles prepared by the SPIDER class. The advantage of this reptile is that the data assembly between the different pages can be solved, but every time the drill climbs a level, you must manually write a function to call crawle, if one The website has a lot of levels of hierarchy, each level page must write an extract function, which must write a large number of extraction functions to splicing extraction, clearly not reach the effect of data automation and splicing. [0004] The second is based on inheriting the crawlspider class, the advantage of this reptile is that the automatic drilling is automatically extracted, and the links of each hierarch...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the technical problem that in the data crawling process in the prior art, the page cannot be automatically drilled down and the target data of all the pages to be crawled can be obtained at the same time

Images