Method and system for grabbing web pages from servers with different IPs (Internet Protocols) in website

Abstract

The invention discloses a method and a system for grabbing web pages from servers with different IPs (Internet Protocols) in a website. The method comprises the following steps of: distributing the IPs of target website servers for the web-page grabbing task of a client side, wherein the web-page grabbing task comprises grabbing of the addresses of the web pages to be grabbed; and then judging whether the web-page grabbing task conforms to the courteous access condition of the servers or not, if SO, utilizing the IP to establish connection with the servers, and grabbing the web pages at the web-page addresses from the servers. In the invention, the access strategy is based on IP level, so that an acquisition working thread is more conveniently controlled to carry out courteous access on the website; by the mode of caching a DNS (Domain Name Server), simultaneously using a plurality of IPs and preferably distributing the fastest IP, the efficiency for grabbing the web pages is greatly improved; and when individual servers of the target website can not be accessed, the servers with other IPs can be switched in time, and the fault-tolerant capability is improved.

Description

technical field [0001] The invention relates to a method and system for grabbing web pages from a website, in particular to a method and system for grabbing web pages from multiple servers with different IPs in the website. Background technique [0002] With the rapid development of the Internet, the scale of information on the Internet is increasing, and the number of website visits is also increasing. In order to meet the current Internet access requirements, most websites with large information scale or large visit volume provide multiple servers with different IPs (Internet Protocol, a protocol for interconnection between networks). Through intelligent DNS (Domain Name System (Domain Name System) server returns a list of server IPs in different orders according to the load balancing strategy, and the client will use the first server to access, thus distributing the user's access requests to different servers. In order to prevent the server from being under too much pres...

AI Technical Summary

Problems solved by technology

For a client, if multiple access requests sent by the client are assigned to the same server of the website, it may be denied service due to the restrictions of the server, or even blocked IP

[0003] Existing crawler systems generally control the access strategy according to the website. Due to the limitation of the number of concurrent website visits, the collection efficiency is low. If the number of working threads for crawling web pages is increased, it is easy to trigger the restricted access conditions of the website, resulting in crawling Failed or blocked IP

Images