Unlock instant, AI-driven research and patent intelligence for your innovation.

Configurable domain name resolution crawler framework and method based on asynchronous HTTP request

A domain name resolution and crawler technology, which is applied in the computer field, can solve problems such as the inability to specify crawling specified computer rooms, the low efficiency of system operation and maintenance personnel, and the inability to ensure that crawlers can traverse all computer rooms, etc., to achieve the effect of improving crawler efficiency

Active Publication Date: 2019-08-16
XIAMEN UNIV TAN KAH KEE COLLEGE
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, most of the frameworks operate on domain names, and cannot specify crawling specified computer rooms, and cannot guarantee that crawlers can traverse all computer rooms
Lead to low work efficiency of system operation and maintenance personnel

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Configurable domain name resolution crawler framework and method based on asynchronous HTTP request

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] Please refer to figure 1 , the present invention provides a configurable domain name resolution crawler framework based on asynchronous HTTP requests, including a domain name resolution control module, a driver module, a persistence module, a link scheduling module, a crawler module and an HTTP communication module;

[0022] In this embodiment, the domain name resolution control is used to control the local domain name resolution results, and the corresponding files will be configured according to the current operating system type;

[0023] The driver module is used to process the data flow of the whole system and control the interaction of data between different components;

[0024] The link scheduling module is used to receive the link sent by the driver module, pack the link into a request object, determine the request sequence of the link ac...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a configurable domain name resolution crawler framework and method based on an asynchronous HTTP request. The configurable domain name resolution crawler framework comprises adomain name resolution control module, a driving module, a persistence module, a link scheduling module, a crawler module and an HTTP communication module. The driving module is respectively linked with the domain name resolution control module and the persistence module box link scheduling module to control data interaction among the impassable components; and the link scheduling module is in data link with the HTTP module. According to the invention, the working efficiency of system operation and maintenance personnel is greatly improved.

Description

technical field [0001] The invention relates to the computer field, in particular to a configurable domain name resolution crawler framework and method based on asynchronous HTTP requests. Background technique [0002] With the increase in the number of visits and the requirements for disaster recovery, the deployment of WEB servers usually develops towards the deployment of multiple computer rooms in different places. This brings a new difficulty in monitoring, how to monitor whether the web services provided by each computer room are normal. For a single computer room, crawlers can be used to crawl all the links of the website, and check the response time, response code, and response content of the links. At present, there are many excellent crawler frameworks in the industry that can realize this function. However, most of the frameworks operate on domain names, and cannot specify crawling specified computer rooms, and cannot guarantee that crawlers can traverse all com...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/41G06F8/71G06F16/951G06F16/955H04L29/08H04L29/12
CPCG06F8/427G06F8/71G06F16/951G06F16/955H04L67/02H04L67/30H04L61/4511Y02D30/50
Inventor 朱喜娜
Owner XIAMEN UNIV TAN KAH KEE COLLEGE