Configurable domain name resolution crawler framework and method based on asynchronous HTTP request

Abstract

The invention relates to a configurable domain name resolution crawler framework and method based on an asynchronous HTTP request. The configurable domain name resolution crawler framework comprises adomain name resolution control module, a driving module, a persistence module, a link scheduling module, a crawler module and an HTTP communication module. The driving module is respectively linked with the domain name resolution control module and the persistence module box link scheduling module to control data interaction among the impassable components; and the link scheduling module is in data link with the HTTP module. According to the invention, the working efficiency of system operation and maintenance personnel is greatly improved.

Description

technical field

[0001] The invention relates to the computer field, in particular to a configurable domain name resolution crawler framework and method based on asynchronous HTTP requests. Background technique

[0002] With the increase in the number of visits and the requirements for disaster recovery, the deployment of WEB servers usually develops towards the deployment of multiple computer rooms in different places. This brings a new difficulty in monitoring, how to monitor whether the web services provided by each computer room are normal. For a single computer room, crawlers can be used to crawl all the links of the website, and check the response time, response code, and response content of the links. At present, there are many excellent crawler frameworks in the industry that can realize this function. However, most of the frameworks operate on domain names, and cannot specify crawling specified computer rooms, and cannot guarantee that crawlers can traverse all com...

Images