Method and equipment for capturing data of website

Abstract

The invention aims to provide a method and equipment for capturing data of a website. The method includes selecting a non-visited link from all links in a current root page according to information of a topological structure of the website and acquiring a next-layer page directed by the non-visited link; judging whether the next-layer page is a target information page or not according to a first preset rule; utilizing the next-layer page as a current root page when the next-layer page is not the target information page, and executing a step a and a step b repeatedly until a first preset condition is met; capturing the target information page when the next-layer page is the target information page; and utilizing a previous root page as a current root page when a second preset condition is met, and executing the step a, the step b, a step c1 and a step c2 repeatedly. Compared with the prior art, the method and the equipment have the advantages that a depth-first traversal mode is adopted, the target data of the integral website can be captured, the target data capturing accuracy is guaranteed, and the data capturing efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a technology for grabbing website data. Background technique [0002] In the prior art, fetching data from data-providing websites generally requires executing a script for each website separately, but when the number of data-providing websites is large, multiple sets of fetching scripts need to be maintained, so script maintenance costs are high, and data fetching The efficiency is not high; at the same time, after the classification information is set on the data providing website, the cookie information of the last classification information will be stored on the server side, but because the traditional data capture generally adopts the breadth-first crawling method, and on the same page When changing the classification information in the website, the uniform resource locator (URL) of the page link will not change, so that after visiting each classification information li...

AI Technical Summary

Problems solved by technology

[0002] In the prior art, fetching data from data-providing websites generally requires executing a script for each website separately, but when the number of data-providing websites is large, multiple sets of fetching scripts need to be maintained, so script maintenance costs are high, and data fetching The efficiency is not high; at the same time, after the classification information is set on the data providing website, the cookie information of the last classification information will be stored on the server side, but because the traditional data capture generally adopts the breadth-first crawling method, and on the same page When changing the classification information in the website, the uniform resource locator (URL) of the page link will not change, so that after visiting each classification information link of the same page, the data that may be captured is the last selected one recorded in the cookie information Classification information, rather than the target data corresponding to each classification information that is expected to be captured, the accuracy of data capture is not high

Images