Web scraping through use of proxies, and applications thereof

Abstract

The present invention relates to a method for securing a web scraping system (1400), the method comprising: at an entry point to the web scraping system (1400), performing the following: (a) validating credentials received with an API request from a client computing device (102), the API request asking that the web scraping system (1400) scrape content from a target website; (b) when the credentials are validated, generating a token (1504) indicating an identity of a client associated with the credentials; (c) transmitting the API request along with the token (1504) to a server (1402A-N, 1404A-N, 1406A-N) configured to initiate a scraping process on the web scraping system (1400); at the server (1402A-N, 1404A-N, 1406A-N) configured to initiate the scraping process on the web scraping system: (d) analyzing the token (1504) to determine whether the client is authorized to conduct the request; and (e) when the client is authorized, causing the web scraping system (1400) to scrape the target website.

Description

The present invention relates to a method for protecting a web scraping system (1400), the method comprising: at an entry point of the web scraping system (1400), performing the following operations: (a) verifying credentials received from a client computing device (102) along with an API request, the API request requesting the web scraping system (1400) to scrape content from a target website; (b) when the credentials are verified, generating a token (1504) indicating the identity of a client associated with the credentials; (c) transmitting the API request along with the token (1504) to servers (1402A-N, 1404A-N, 1406A-N) configured to initiate a scraping process on the web scraping system (1400); at the servers (1402A-N, 1404A-N, 1406A-N) configured to initiate the scraping process on the web scraping system: (d) Analyze the token (1504) to determine whether the client is authorized to make the request; and (e) when the client is authorized, cause the web crawling system (1400) to crawl the target website.

Claims

1. A computer-implemented method for securing a web scraping system (1400), the method comprising: at an entry point to the web scraping system (1400), performing the following: (a) validating credentials received with an API request from a client computing device (102), the API request asking that the web scraping system (1400) scrape content from a target website; (b) when the credentials are validated, generating a token (1504) indicating an identity of a client associated with the credentials; (c) transmitting the API request along with the token (1504) to a server (1402A-N, 1404A-N, 1406A-N) configured to initiate a scraping process on the web scraping system (1400); at the server (1402A-N, 1404A-N, 1406A-N) configured to initiate the scraping process on the web scraping system: (d) analyzing the token (1504) to determine whether the client is authorized to conduct the request; and (e) when the client is authorized, causing the web scraping system (1400) to scrape the target website.

2. The method of claim 1, further comprising: (f) passing the API request between a plurality of servers (1402A-N, 1404AN, 1406A-N), each configured to perform a function of the web scraping system (1400), the server (1402A-N, 1404A-N, 1406A-N) configured to initiate the web scraping system being included in the plurality of servers (1402A-N, 1404A-N, 1406A-N); at each of the respective servers (1402A-N, 1404A-N, 1406A-N): (g) analyzing the token to (1504) determine whether the client is authorized to conduct the function performed by the respective server (1402A-N, 1404A-N, 1406A-N); and (e) when the client is authorized to conduct the function, performing the function.

3. The method of claim 2, wherein the plurality of servers (1402A-N, 1404A-N, 1406A-N) includes a server (1402A-N) configured to service API requests formatted as a web proxy request.

4. The method according to any one of the claims 2 and 3, in particular according to claim 2, wherein the plurality of servers (1402A-N, 1404A-N, 1406A-N) includes a server (1404A-N) configured to service synchronous API requests, leaving a connection between the web scraping system (1400) and the client computing device (102) open while the web scraping system (1400) scrapes the target website.

5. The method according to any one of the claims 2 to 4, in particular according to claim 2, wherein the plurality of servers (1402A-N, 1404A-N, 1406A-N) includes a server (1406A-N) configured to service asynchronous API requests, closing a connection between the web scraping system (1400) and the client computing device before the web scraping system (1400) scrapes the target website.

6. The method according to any one of the claims 2 to 5, in particular according to claim 2, wherein the generating (b) comprises generating the token (1504) to include a role of the client, the role indicating whether a particular function of the web scraping system (1400) is authorized.

7. The method according to any one of the claims 2 to 6, in particular according to claim 2, wherein the generating (b) comprises generating the token (1504) to include a digital signature (1514) that cryptographically guarantees that the identity of the client has not been tampered with.

8. The method according to any one of the claims 2 to 7, wherein the API request is a first API request, and the token (1504) is a first token, further comprising: (f) validating credentials received with a second API request, the second API request asking to retrieve content that the web scraping system (1400) has previously scraped from the target website; (g) when the credentials are validated, generating a second token indicating an identity of a client associated with the credentials received with the second API request; (h) determining whether the first (1504) and second tokens indicate that the first and second API requests came from the client; and (i) when the first (1504) and second tokens indicate that the first and second API requests came from the client, returning the scraped content in response to the second API request.

9. The method of claim 8, further comprising, when the first (1504) and second tokens do not indicate that the first and second requests came from the client, refusing to return the scraped content.

10. The method according to any one of the claims 2 to 9, wherein the entry point is a load balancer (1208A, 1208B, 1208C) that selects the server (1402A-N, 1404A-N, 1406A-N) from a plurality of parallel servers (1402A-N, 1404A-N, 1406A-N).

11. The method according to any one of the preceding claims, wherein the plurality of servers (1402A-N, 1404A-N, 1406A-N) includes a first server (1402A-N) configured to service API requests formatted as a web proxy request and configured to reformat the request according to an API with known parameters and to send the reformatted request to at least one second server (1404A-N), the second server being configured to service synchronous API requests.

12. The method according to any one of the preceding claims, in particular according to claim 11, wherein the second server (1404A-N) is adapted to send the API request to at least one third server (1404A-N), the third server (1406A-N) being configured to service asynchronous API requests.

13. The method according to any one of the preceding claims, in particular according to claim 12, wherein the third server (1406A-N) is adapted to generate a job description and to send the job description to a request status manager (304) for initiating the scraping job.

14. The method according to any one of the preceding claims, wherein a payload of the token (1504) includes an expiration date or time for the token (1504).

15. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform the method of any one of the preceding claims.

Images