An Object Storage Based Crawler Network Path Tracing Method

Abstract

The invention discloses a crawler network path tracing method based on object storage. The method comprises the steps that an object storage system and a log recorder are established, wherein the log recorder generates a result path log, and indexes from the source URL of a crawling result to a crawler result file on the object storage system are recorded in the result path log; when an external system needs to call the data in the database, the crawler result file on the object storage system is directly obtained through the indexes. According to the method, the object storage system is introduced so that the file reading and writing speed can be increased; the result path log is established so that the data can be retrieved in the log when the external system calls the data and does not need to be searched in the database, and accordingly the possibility of reading and writing conflicts is avoided.

Description

technical field

[0001] The invention belongs to the research field of path tracing in software engineering, in particular to a crawler network path tracing method based on object storage. Background technique

[0002] A web crawler is a program or script that automatically captures information on the World Wide Web according to certain rules. In the current path tracing, most of the crawler network path tracing is based on the crawler task as the basic unit. For example, the open source crawler framework pyspider, the default The action is to store the result into the database. If the external system needs to retrieve the data in the database, there is no convenient retrieval method. It can only scan the database, and it is necessary to modify the status of the result data in the database so that these processed data will be excluded in the next processing. result. As a result, the data in the database needs to be maintained by the two systems together, causing great uncer...

Images