Extensible distributed data acquisition method and system

Abstract

The invention relates to an extensible distributed data acquisition method and system. The method comprises the following steps that a main node, a working node and an intermediate node are deployed;the main node regularly generates an acquisition task according to a timing task in a database and publishes the acquisition task to a task queue of a message queue; the working node reads the collection task from the task queue at regular time and decides whether to apply for executing the collection task according to the state of the local server; the main node selects an optimal working node from the working nodes applying for executing the same acquisition task to execute the acquisition task, and removes the acquisition task from the task queue; the working node generates and executes anacquisition process according to the acquisition task, and puts the acquired data into a data queue in a message queue of the intermediate node; and the working node monitors the running state of theacquisition process and records related data. According to the invention, the contradiction among usability, universality and manageability of the existing data acquisition system is solved, and the problem of unreasonable task allocation in distributed deployment is solved.

Description

Technical field [0001] The present invention relates to the field of computers, and in particular to a scalable distributed data collection method and system. The data collection mentioned in the present invention specifically refers to data collection publicly available on the Internet. Background technique [0002] Network data collection refers to the process of obtaining public data resources from the Internet and then saving them to a specific location as required. Network data collection is usually realized by using a web crawler, which is a computer program that automatically crawls Internet web page data according to certain rules. A crawler can download webpages from a certain website on the Internet, then parse and filter the webpages, and extract data from them as required. With the development of artificial intelligence and big data technology, the demand for network data is increasing rapidly. A research job requires tens of millions of data, which has become very c...

AI Technical Summary

Problems solved by technology

[0006] The present invention proposes a highly scalable distributed data acquisition framework and method, which solves the contradiction between the ease of use, versatility and manageability of existing data acquisition systems and the problem of unreasonable task allocation in distributed deployment

Images