Highly-decoupled method capable of dynamically managing crawlers

Abstract

The invention discloses a highly-decoupled method capable of dynamically managing a crawler, which comprises the following steps that: the crawler is divided into two stages of data analysis and new target generation, rules of the two stages corresponding to a collection target are compiled into json data according to a protocol, and the json data are stored into a host end; the host runs a task and sends the task to a client with sufficient resources through a message queue module according to a resource scheduling algorithm, the client receives task information, converts the task information into executable information through the crawler protocol core, runs the executable information through a crawler running module, and finally obtains data; the host end obtains data and a new task, and stores and updates a task pool; and the host end is separated from a crawler server, so that the coupling of the system can be reduced. Therefore, after the functions of the crawler are separated, the complexity of the crawler server can be reduced, and the host end can be modified while the distributed crawler system runs so as to achieve the purpose of specific control management, so that the whole module is subjected to decoupling and extensible design, and the robustness and stability of the whole framework are enhanced.

Description

technical field [0001] The invention relates to computer data mining technology, in particular to a highly decoupled and dynamically managed crawler method. Background technique [0002] As a tool for network information search, search engine collects and discovers information on the Internet with certain strategies, understands, extracts, organizes and processes information, and provides retrieval services for users. In 1994, the crawler program was applied to the indexing program, and Yahoo, Google, etc. appeared one after another. But so far, no matter how powerful the search engine is, there are still problems such as information loss, low update rate, and low accuracy rate. Users need faster, more accurate, more convenient, and more effective query services, which has become the goal of research and development of search engine technology. [0003] In this case, topic crawlers that directionally grab related web resources came into being. Theme crawler, also known as...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the current problem of coupling phenomena in the realization of functions such as data capture, data analysis, data scheduling, and updating, the method proposed by the present invention, which is suitable for highly decoupled and dynamically manageable crawlers, can be based on effective data scheduling and updating methods To achieve the corresponding decoupling phenomenon

Images