A highly decoupled method for dynamically managing reptiles

Abstract

The invention discloses a highly decoupled and dynamically managed crawler method. The method divides the crawler into two stages: data analysis and new target generation, and writes the rules corresponding to the two stages of acquisition targets into json data and stores them in the host computer according to the protocol. end; the host end runs the task, sends it to the client with sufficient resources through the message queue module according to the resource scheduling algorithm, and the client receives the task information, converts it into executable information through the crawler protocol core and runs it by the crawler running module, and finally obtains the data; The host end obtains data and new tasks, stores and updates the task pool; separating the host end from the crawler server can reduce the coupling of the system. Therefore, after the crawler function is separated, the complexity of the crawler server can be reduced, and the host side can be modified while the distributed crawler system is running to achieve specific control and management purposes, thereby decoupling and extensible the entire module Design to enhance the robustness and stability of the entire framework.

Description

technical field [0001] The invention relates to computer data mining technology, in particular to a highly decoupled and dynamically managed crawler method. Background technique [0002] As a tool for network information search, search engine collects and discovers information on the Internet with certain strategies, understands, extracts, organizes and processes information, and provides retrieval services for users. In 1994, the crawler program was applied to the indexing program, and Yahoo, Google, etc. appeared one after another. But so far, no matter how powerful the search engine is, there are still problems such as information loss, low update rate, and low accuracy rate. Users need faster, more accurate, more convenient, and more effective query services, which has become the goal of research and development of search engine technology. [0003] In this case, topic crawlers that directionally grab related web resources came into being. Theme crawler, also known as...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the current problem of coupling phenomena in the realization of functions such as data capture, data analysis, data scheduling, and updating, the method proposed by the present invention, which is suitable for highly decoupled and dynamically manageable crawlers, can be based on effective data scheduling and updating methods To achieve the corresponding decoupling phenomenon

Images