Network content asynchronous grasping system and method

Abstract

The invention proposes a network asynchronous grasping system and method. The network asynchronous grasping system comprises a task queue manager used for providing at least one task queue; a scheduler used for reading a uniform resource locator URL of a network content to be grasped from each task queue, and triggering a driver to schedule the URL according to the environment type of a back end where a task to which the URL belongs locates; the driver used for reading task information of the task of the URL after being triggered by the scheduler, injecting the URL into a grasping pool based on the task information, and controlling the frequency of injecting the URL into the grasping pool according to the task information, wherein the task information comprises the query per second and a concurrency value; and an actuator used for reading the URL from the grasping pool, and grasping the URL. The invention can ensures the stability of the grasping system during high concurrency, effectively save system resources and improve the grasping performance.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a system and method for asynchronously capturing network content. Background technique [0002] With the development of the Internet, the Internet will contain a large amount of network content. In some application scenarios, it is necessary to use some computer technology to extract the network content that users need from the massive network content. This computer technology is called crawling. For example, web content can be crawled by using a crawler. [0003] In related technologies, the grabber adopts a concurrency control strategy, or a query rate per second (Query Per Second, QPS) control strategy, wherein, the concurrency control strategy independently controls the total concurrent queue length through threads or processes, and each process or Threads execute fetching synchronously to ensure that the total length of the queue is fixed and the pressure on the system...

AI Technical Summary

Problems solved by technology

[0004] Under these two methods, the control granularity is too extensive. For slow back-end systems, the crawling performance is poor, and the stability of network content capture cannot be fully guaranteed, which may easily cause an avalanche effect of the crawling system.

Images