Web scraping through use of proxies, and applications thereof
The method of using multiple independent database servers with health checks addresses inefficiencies in managing server health during web scraping, enhancing reliability and efficiency.
Patent Information
- Authority / Receiving Office
- HK · HK
- Patent Type
- Applications
- Current Assignee / Owner
- OXYLABS UAB
- Filing Date
- 2026-05-27
- Publication Date
- 2026-07-10
AI Technical Summary
Existing web scraping technologies face inefficiencies in managing and monitoring the health of multiple database servers during web scraping jobs, leading to potential failures and suboptimal performance.
A computer-implemented method utilizing multiple independent database servers to manage data storage for web scraping jobs, with health checks to determine server enablement and disablement, ensuring efficient operation and reliability.
Enhances the reliability and efficiency of web scraping operations by ensuring only healthy database servers are utilized, thereby reducing failures and improving overall performance.
Smart Images
Figure 00000001_0000 
Figure 00000053_0000 
Figure 00000054_0000
Abstract
Description
Invention Title: Web Scraping Using Proxy and Its Application Summary: This invention relates to a computer-implemented method for processing web scraping jobs, using multiple database servers (404A-404N) that operate independently of each other, and each database server is configured to manage data storage to at least a portion of a job database (314), the job database storing the state of the web scraping job at the time of execution of the web scraping job, the method comprising: receiving a web scraping request from a client computing device (102); and, upon receiving the web scraping request, selecting from the multiple database servers (404A-404N) identified as being in a table (1008) A database server is enabled; and a job description specified by the web crawling request is sent to the selected database servers (404A-404N) to be stored as a pending web crawling job in the job database (314); the health of each of the plurality of database servers (404A-404N) is repeatedly checked; and based on the health check, it is determined whether each of the plurality of database servers (404A-404N) is enabled or disabled in table (1008). Summary
Claims
1. A computer-implemented method for processing web scraping jobs, using a plurality of database servers (404A-404N) operating independently of one another and each being configured to manage data storage to at least a portion of a job database (314) that stores status of web scraping jobs while the web scraping jobs are being executed, the method comprising: - receiving a web scraping request from a client computing device (102); - when the web scraping request is received, selecting one of the plurality of database servers (404A-404N) that is identified as enabled in a table (1008); and sending a job description specified by the web scraping request to the selected database server (404A-404N) for storage in the job database (314) as a pending web scraping job; - repeatedly checking health of each of the plurality of database servers (404A-404N); and - based on the health checks, determine whether each of the plurality of database servers (404A-404N) are to be enabled or disabled in the table (1008); characterized in that each of the plurality of database servers (404A-404N) comprises a message broker (454) that queues job descriptions to be stored in the jobs database (314), and each of the repeatedly checking comprises, for each of the plurality of database servers (404A-404N), - checking a connection between a server (302) that receives web scraping requests from client computing devices (102) and the respective database server's message broker (454); and / or - checking a number of messages queued within the respective database server's message broker (454).
2. The method of claim 1, wherein each of the repeated checking comprises, for each of the plurality of database servers (404A-404N), connecting to the portion of the job database (314) for the respective database server (404A-404N).
3. The method of any one of the preceding claims, wherein by checking health, it is determined whether the respective database server (404A-404N) is available to accept new web scraping jobs.
4. The method of any one of the preceding claims, wherein each of the plurality of database servers (404A-404N) is marked as disabled in the table (1008) if the connection between the server (302) and the respective database server's message broker (454) is nonresponsive or returns an error message.
5. The method of any one of the preceding claims, wherein each of the plurality of database servers (404A-404N) is marked as disabled in the table (1008) if the number of messages exceeds a threshold.
6. The method of any one of the preceding claims, wherein, each of the plurality of database servers (404A-404N) that is, based on the health checking, determined to be overworked or hung-up, is marked for read-only access in the table (1008).
7. The method of any one of the preceding claims, further comprising - consuming, by a database microservice (456) of the selected database server (404A-404N), the job description specified by the web scraping request; and - initiating processing of the web scraping job.
8. The method according to any of the preceding claims, wherein each of the plurality of database servers (404A-404N) is a shard managing storage in a horizontal partition of the jobs database (314).
9. The method according to any of the preceding claims, wherein each of the plurality of database servers (404A-404N) do not synchronize states to one another.
10. The method according to any of the preceding claims, wherein the plurality of database servers (404A-404N) are executed by a plurality of different computing devices.
11. The method according to any of the preceding claims, further comprising: determining whether a number of database servers that are disabled in the plurality of database servers exceeds a threshold; and when the number of database servers that are disabled exceeds the threshold, alerting an administrator.
12. The method according to any one of the preceding claims, further comprising: - retrieving a next value from a counter maintained by the selected database server (402A-404N); - concatenating an identification of the selected database server (402A-404N) with a timestamp and the next value to generate a job identifier associated with the web scraping job.
13. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations, comprising the steps of any one of the preceding claims.
14. A system for processing web scraping jobs, comprising: a processor; a job database (314) that stores the status of web scraping jobs while the web scraping jobs are being executed; a memory that stores the job database (314); a plurality of database servers (404A-404N) operating independently of one another, each database server (404A-404N) configured to manage data storage to at least a portion of the job database (314); a database monitor configured to repeatedly check health of each of the plurality of database servers (404A-404N) and, based on the results of the health checks, determine whether each of the plurality of database servers (404A-404N) are to be enabled or disabled in a table (1008); a database server selector configured to, when a web scraping request is received from a client computing device (103), select one of the database servers (404A-404N) identified as enabled in the table; and a request intake manager (302) configured to send a job description specified by the web scraping request (302) to the selected database server (404A-404N) for storage in the job database (314) as a pending web scraping job, characterized in that each of the plurality of database servers (404A-404N) comprises a message broker (454) that queues job descriptions to be stored in the jobs database (314), and the database monitor is configured to, for each of the plurality of database servers (404A-404N), - check a connection between a server (302) that receives web scraping requests from client computing devices (102) and the respective database server's message broker (454); and / or - check a number of messages queued within the respective database server's message broker (454).
15. The system of claim 14, wherein the database monitor is configured to, for each of the plurality of database servers (404A-404N), check a connection between the request intake manager (302) and the jobs database (314).