A load balancing scheduling and abnormal retry method for a web crawler proxy tunnel
Through load balancing scheduling and exception retry methods, the problems of low crawling efficiency and resource waste of network crawlers under high concurrency and exceptional conditions are solved, and the crawling speed and cost savings are achieved.
Patent Information
- Application Number
- CN202010019709.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-01-08
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-01-08
AI Technical Summary
Existing network crawlers tend to cause proxy tunnels to deny access in high concurrency, which increases costs, and simple retry in exceptional situations will lead to long task time and wasted proxy resources.
Through load balancing scheduling and exception retry methods, the database records the successful and failed URLs, start the crawler regularly, select and use them according to the weight of the proxy tunnel, and give priority to crawling URLs with high failures, and record and Try again to avoid high concurrency and resource waste.
It improves the crawling speed of network crawlers, reduces the failure rate, realizes balanced utilization of proxy tunnels, and saves costs.
Smart Images

Figure CN111240839B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of network search, and in particular relates to a load balancing scheduling and abnormal retry method for a network crawler proxy tunnel. Background Art
[0002] With the increasing growth of network information, web crawlers have become an important part of information processing. A web crawler is a program or script that automatically obtains web page information according to certain rules. Web crawlers usually reside on servers.
[0003] Currently, web crawlers generally need to use proxies, and proxy tunnels are one of the best solutions. Proxy tunnels generally have high concurrency limits. If the number of concurrent accesses to a web page exceeds the specified value, the proxy tunnel will deny access and return an error code. The solution to high concurrency is generally to spend money to buy a proxy tunnel with a higher concurrency limit, but this will increase costs.
[0004] It is generally impossible for a web crawler to successfully crawl all target web pages at once, because there may be various abnormal situations such as: network abnormality, network slowness, target web page loss, slow target web page access speed, proxy abnormality, etc. If you simply try to crawl the entire page multiple times, the crawling task may take a long time to complete, and there will be a certain waste of proxy resources. Summary of the invention
[0005] In view of the above problems, the present invention provides a load balancing scheduling and abnormal retry method for a web crawler proxy tunnel.
[0006] A load balancing scheduling and abnormal retry method for a web crawler proxy tunnel comprises the following steps:
[0007] S1. Start the web crawler in a normal crawling mode, and terminate the process of the crawler running in the current mode before starting;
[0008] S2. Start each web crawler at a fixed time within a day. The start time of each web crawler is different to avoid starting a large number of web crawlers in a certain period of time, which will lead to an increase in the high concurrency of proxy tunnels.
[0009] S3, sorting according to the weight of each proxy tunnel, and the web crawler selects a proxy tunnel to use according to the weight;
[0010] S4, recording in the database the URLs that each web crawler failed to crawl and succeeded to crawl;
[0011] S5. Within a certain period of time (e.g., 1 hour), use a unified database to record the number of times each proxy tunnel is used;
[0012] S6. Within a certain period of time (e.g., 1 hour), use a unified database to record the number of failures of each proxy tunnel due to high concurrency;
[0013] S7. Different weights are set according to the number of failures of each proxy tunnel due to high concurrency, where the proxy tunnel with a large number of failures has a lower weight, and the proxy tunnel with a small number of failures has a higher weight;
[0014] S8. Same as the normal crawling method, start the web crawler regularly every day in an abnormal retry mode;
[0015] S9, start in abnormal retry mode, the start time is a certain period of time after the normal crawling start, and before starting, the crawler process running in the current mode is terminated;
[0016] S10, start the web crawler in an abnormal retry mode, read the URLs that failed to crawl from a unified database, and give priority to crawling the URLs with the highest number of failures;
[0017] S11, record the successfully crawled web pages in the database, and directly skip the successfully crawled web pages and no longer crawl them;
[0018] S12, crawling by abnormal retry mode, judging whether it is a home page, a list page or a detailed content page according to the characteristics of the URL, and crawling them respectively;
[0019] S13. If crawling the web page in an abnormal retry manner fails again, the number of failures for the URL is recorded;
[0020] S14. Repeat steps S8 to S13 until there are no URLs recorded as failed crawling in the database.
[0021] Furthermore, the crawling of the failed URLs by retrying multiple times includes:
[0022] Record every URL that the web crawler failed to crawl in a unified database, excluding pages with a 404 status;
[0023] Record each URL that the web crawler successfully crawls in a unified database;
[0024] Restart the web crawler regularly to re-crawl the URLs that are recorded as crawling failures.
[0025] Furthermore, the periodic restarting of the web crawler to re-centrally crawl the URLs recorded as crawling failures also includes:
[0026] According to the time interval between the start time of normal crawling of the web crawler and the start time of abnormal retry, the web crawler starts crawling in abnormal retry mode after a certain period of time of normal crawling.
[0027] The URLs to be crawled are read from a unified database.
[0028] Compared with the prior art, the invention has the following beneficial effects: the invention can improve the crawling speed of the web crawler and reduce the failure rate. The invention realizes the balanced utilization of the proxy tunnel in the algorithm, greatly avoids the waste of the proxy tunnel and saves the cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a step diagram of the present invention;
[0030] Figure 2 It is a flow chart of load balancing scheduling of a web crawler proxy tunnel of the present invention;
[0031] Figure 3 It is a flow chart of the abnormal retry method of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] The embodiment of the present invention provides a load balancing scheduling and abnormal retry method for a web crawler proxy tunnel, and the specific steps are as follows:
[0034] S1. The web crawler startup method of the present invention is divided into two methods: normal crawling and abnormal retry. The crawler is started in the normal crawling method first, and the process of the crawler running in the current mode is terminated before the startup;
[0035] S2. Setting timers for starting and ending the web crawler so that it can be automatically executed at a fixed time;
[0036] S3, multiple web crawlers are distributed and started at different times. After starting one web crawler, another one is started after a certain period of time to avoid starting all of them at once, which may lead to too high a concurrent number of proxy tunnels in a certain period of time.
[0037] S4, each web crawler selects a proxy tunnel to use according to the weight;
[0038] S5, before crawling each web page, query the URL from the unified database. If the URL has been successfully crawled, skip crawling;
[0039] S6. Record the URLs that failed to be crawled into a unified database. Do not record the web pages with 404 status (webpage does not exist) and 500 status (server error).
[0040] S7, recording the successfully visited website into a unified database, and if it already exists, overwriting and updating the modification time;
[0041] S8. After a certain period of time, the web crawler is started in the normal crawling mode and then started in the abnormal retry mode. Before starting, the crawler process currently running in the current mode is terminated. At any time, there can be at most one process started in the normal crawling mode and one process started in the abnormal retry mode;
[0042] S9, the abnormal retry mode is started, the URLs that failed to be crawled are read from the unified database, and then the crawling begins;
[0043] S10, abnormal retry mode is started, different methods of extracting web page information are used for different URLs, and the URL is judged according to its characteristics whether it belongs to the detailed information page, list page or home page, and crawled respectively;
[0044] S11. If crawling the web page in an abnormal retry manner fails again, the number of failures for the URL is recorded;
[0045] S12. Repeat steps S8 to S11 until there are no URLs recorded as failed crawling in the database.
[0046] Among them, S6 records the URLs that failed to be crawled, and the number of failures needs to be recorded. When crawling web pages in abnormal retry mode, the one with the highest number of failures will be crawled first;
[0047] Record the usage count of each proxy tunnel in each time period (e.g., 1 hour) in a unified database;
[0048] Record the number of web page crawling failures caused by high concurrency in each proxy tunnel in each time period (e.g., 1 hour) in a unified database;
[0049] S4 selects a proxy tunnel to use, with the weight initialized to 0. Then, it needs to sort the proxy tunnels according to the number of webpage crawling failures caused by high concurrency in the previous time period, and set different weights. The higher the number, the lower the weight, and the lower the number, the higher the weight. According to the weight, the proxy tunnel with the highest weight is selected to achieve the purpose of proxy tunnel load balancing.
[0050] S6 records the URLs that failed to crawl into a unified database. When retrying abnormally, these URLs that failed to crawl are read from the unified database and then crawled, instead of repeatedly crawling from the starting web page of the site. There are two ways to start the web crawler of the present invention, one is normal startup, and the other is abnormal retry startup. When starting the web crawler, parameters will be added to the command line. Normal startup can bring the --type=default parameter after the execution command line, and abnormal retry startup can bring the --type=exception parameter after the execution command. After the crawler is started, it will determine whether it needs to crawl from the starting web page of the site, or read the URLs marked as failed to crawl from the unified database and then crawl.
[0051] S7 records the successfully visited URLs into a unified database. When crawling in abnormal retry mode, it will not crawl the successfully crawled web pages again.
[0052] The above is a specific description of the implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for load balancing scheduling and abnormal retry of a web crawler proxy tunnel, comprising the following steps: S1. Start the web crawler in a normal crawling mode, and terminate the process of the crawler running in the current mode before starting; S2. Start each web crawler at a fixed time within a day. The start time of each web crawler is different to avoid starting a large number of web crawlers in a certain period of time, which will lead to an increase in the high concurrency of proxy tunnels. S3, sorting according to the weight of each proxy tunnel, and the web crawler selects a proxy tunnel to use according to the weight; Set different weights. The weight of a higher number of times is lower, and the weight of a lower number of times is higher. According to the weight, the proxy tunnel with the highest weight is selected to achieve the purpose of proxy tunnel load balancing; S4, recording in the database the URLs that each web crawler failed to crawl and succeeded to crawl; S5. Use a unified database to record the number of times each proxy tunnel is used within a certain period of time; S6. Use a unified database to record the number of failures of each proxy tunnel due to high concurrency within a certain period of time; The URLs that failed to be crawled are recorded in a unified database. When retrying abnormally, these URLs that failed to be crawled are read from the unified database and then crawled, instead of repeatedly crawling from the starting page of the site. There are two ways to start the web crawler, one is normal start, and the other is abnormal retry start. When starting the web crawler, parameters will be added to the command line. Normal start will add the --type=default parameter after the command line execution, and abnormal retry start will add the --type=exception parameter after the command execution. After the crawler is started, it will determine whether to crawl from the starting page of the site or read the URLs marked as failed crawling from the unified database and then crawl them according to the type parameter; S7. Different weights are set according to the number of failures of each proxy tunnel due to high concurrency, where the proxy tunnel with a large number of failures has a lower weight, and the proxy tunnel with a small number of failures has a higher weight; S8. Start the web crawler regularly every day in the abnormal retry mode, just like the normal crawling mode. At any time, there can be only one process started by the normal crawling mode and one process started by the abnormal retry mode. S9, start in abnormal retry mode, the start time is a certain period of time after the normal crawling start, and before starting, the crawler process running in the current mode is terminated; S10, start the web crawler in an abnormal retry mode, read the URLs that failed to crawl from a unified database, and give priority to crawling the URLs with the highest number of failures; S11, record the successfully crawled web pages in the database, and directly skip the successfully crawled web pages and no longer crawl them; S12, crawling by abnormal retry mode, judging whether it is a home page, a list page or a detailed content page according to the characteristics of the URL, and crawling them respectively; S13. If crawling the web page in an abnormal retry manner fails again, the number of failures for the URL is recorded; S14. Repeat steps S8 to S13 until there are no URLs recorded as failed crawling in the database.
2. The method according to claim 1, characterized in that The step S6 of retrying crawling of the URLs that failed to be crawled multiple times includes: Record every URL that the web crawler failed to crawl in a unified database, excluding pages with a 404 status; Record each URL that the web crawler successfully crawls in a unified database; Restart the web crawler regularly to re-crawl the URLs that are recorded as crawling failures.
3. The method according to claim 2, characterized in that The periodic restart of the web crawler to re-centrally crawl the URLs recorded as crawling failures also includes: According to the time interval between the start time of normal crawling of the web crawler and the start time of abnormal retry, the web crawler starts crawling in abnormal retry mode after a certain period of time of normal crawling. The URLs to be crawled are read from a unified database.
Citation Information
Patent Citations
Redirection method for virtual machine network connection when on-line migrating striding sub network
CN101465812A
Shared memory for intelligent network interface cards
CN108696461A
A method and a device for detecting the running state of a network crawler
CN109298987A
system and method for detecting and reporting online activity using real-time content-based network monitoring
US20020128925A1
