Distributed network data acquisition method and acquisition system thereof

Abstract

The invention relates to the technical field of network data acquisition, in particular to a distributed network data acquisition method and an acquisition system thereof. The method comprises the following steps: eliminating duplications on a link in a request queue by virtue of a scheduler, and assigning the request queue to a corresponding slave node and performing network data acquisition; when a network data acquisition behavior of a certain acquisition node encounters an attack behavior of an acquired website, triggering a corresponding defense mechanism; judging an attack type accordingto the attack behavior by the defense mechanism, and judging whether the attack type is matched with a preset defense type of the slave node corresponding to the acquisition node or not; if yes, executing a defense measures corresponding to the defense type to eliminate an attack; if not, cancelling the network data acquisition behavior of the acquisition node, and returning a request queue not acquired to the scheduler for waiting for reassignment, so that a corresponding measures can be timely adopted for eliminating a crisis when normal network data acquisition work encounters an error attack of the acquired website.

Description

Technical field [0001] The invention relates to the technical field of network data collection, in particular to a distributed network data collection method and its collection system. Background technique [0002] Network data collection refers to the use of Internet search engine technology to achieve targeted, industry-specific and accurate data capture, and to classify data according to corresponding rules to form a database file. [0003] The patent publication number CN108121706A is a distributed crawler optimization method. The specific steps of the distributed crawler optimization method are as follows: the dispatch center issues tasks; the crawler grabs the content of the webpage according to the URL; the parser parses the content of the webpage; If there are more updates, the content of the web page is returned to the data warehouse; the parser parses the links in the web page and uses the Bloom filter locally to de-duplicate; hash the URLs that have passed the local de-d...

AI Technical Summary

Problems solved by technology

[0005] When collecting data information on agricultural websites, although the crawler work complies with the Robots protocol to interact with the website, the long-term and / or frequent normal crawler work may be attacked by the website's anti-crawler error, and cannot be performed normally. fetching jobs for

Images