A method for using large-scale IP address resources in a targeted information capture scenario

Abstract

The invention provides a method for using large-scale IP address resources in a directional information grabbing scene. The method includes the following steps: for the network resources with access frequency restrictions set, establish a priority queue including all IP addresses in the IP address set L; when allocating available IP addresses, take out the IP address with the highest priority in the priority queue, and update the IP address The time when the network resource can be accessed next time, if the current time is greater than or equal to t, the current task can use the IP address immediately; for each IP address, maintain the number of times un and access failures fn of the IP address, when from When the IP address with the highest priority is taken out from the priority queue, it is selected with a probability of 1‑fn / un, and discarded with a probability of fn / un. It not only realizes the full use of IP address access capabilities, but also improves the success rate of network information acquisition tasks.

Description

technical field [0001] The present invention relates to the field of specific network information acquisition, and in particular to a method for using large-scale IP address resources in a directional information capture scenario, which can be used efficiently when the access frequency of a single IP to specific network resources is limited and a large number of tasks are executed concurrently And allocate a large number of IP address resources. Background technique [0002] With the rapid development of the Internet, the data resources on the network are also expanding rapidly. In some scenarios that require centralized acquisition of network resources, such as search engine crawlers crawling the web pages of a certain website, due to the huge number of target web pages, simply executing the acquisition tasks in a single thread serially is far from achieving performance. Require. The more commonly used method at this time is to execute multiple information acquisition tas...

AI Technical Summary

Problems solved by technology

[0004] At the same time, in actual scenarios, in order to save crawling costs, a large number of IP addresses are usually public proxies (HTTP or Socks proxies) on the Internet. The quality and stability of the proxies are usually uncontrollable, and even poor overall availability may occur. If it is used indiscriminately during the crawling process, it will cause a large number of unnecessary access exceptions

Images