Dynamic optimization of request parameters for proxy servers

Abstract

The system and method of task fulfillment are extended as provided herein and target the web scraping process by the step of the client submitting a request to the web crawler. The system and method allow for defining more complex requests for the web crawler in order to receive more specific data. In one aspect, a method for extracting and collecting data from a network by a service provider infrastructure comprises the steps of: checking parameters of a request received from a user's device, adjusting the request parameters according to pre-established scraping logic, selecting a proxy according to criteria of the pre-established scraping logic, sending the adjusted request to a target through the selected proxy, checking metadata received from the target, and forwarding the data to the user's device.

Description

Technical Field

[0001] In one aspect, the embodiments detailed herein relate to retrieving data from the web to improve communication between users and technologies used to access publicly available internet resources. The embodiments detailed herein utilize data collection techniques to more efficiently collect large amounts of data from public web resources. Background Technology

[0002] A proxy server is an intermediary server that accepts requests from clients and forwards them to other proxy servers, origin servers, or services from their own caches. A proxy server is a computing device connected to a network that acts as an intermediary for client computing devices, requesting services or data from "network resources," accepting requests from client proxy servers, and either forwarding requests to another proxy, redirecting them to the requested resource, or serving requests from their own caches located locally, remotely, or distributed across multiple systems.

[0003] A proxy is also known as a "server" or "gateway." A proxy allows users on a network to browse the web, send files via File Transfer Protocol (FTP), and work using email and other internet services. In one aspect, a proxy operates by: i) receiving requests from users on public networks or private networks separated by firewalls, containing desired content from remote servers; ii) creating a proxy tunnel to the remote server; iii) forwarding the request to the remote server through the established tunnel; iv) receiving responses from the remote server; and v) forwarding the responses back to the user.

[0004] Depending on the functionality provided or the server used, proxies can be categorized into different types. Proxies can also be classified as residential Internet Protocol (IP) proxies, data center IP proxies, and mobile IP proxies. A residential IP proxy is an IP address linked to a physical device (such as a mobile phone or desktop computer). The actual owner of a residential IP proxy, the Internet Service Provider (ISP), registers the residential IP address in a public database, which allows websites to determine the device's Internet provider, network, and location. A data center IP proxy is a proxy server assigned a data center IP address. Data center IPs are owned by companies, not individuals. Data center proxies do not actually have IP addresses located in a home. Unlike residential proxies, data center proxies are associated with commercial data centers. Mobile IP proxies behave similarly to residential proxies because their IP addresses are assigned by mobile operators from a pool of IP addresses assigned to mobile clients. Mobile IP proxies use mobile data, unlike residential proxies which use broadband ISP connections.

[0005] A proxy server is essentially a computer on the internet that has its own IP address, which the client's computer "knows." When a client sends a web request, the request is first sent to the proxy server. The proxy server then makes the web request on behalf of the client, collects the response from the web server, and forwards the web page data so that the client can view the page in a browser or in another way (e.g., as data written to a file on a disk for batch processing by data aggregation and analysis tools). When the proxy server forwards the web request, it can modify the data but still provide the requested data. For example, the proxy server can change the client's IP address, thus not providing the web server with the client's geographical location. Furthermore, the proxy server can block access to certain web pages based on IP address or domain name.

[0006] Modern proxy servers do more than simply forward web requests. For example, they can perform additional functions to improve data security and network performance. Proxy servers can act as firewalls and web filters, provide shared network connectivity, and cache data to speed up common requests. They can offer a high level of privacy. Proxy servers can alter the IP address and other identifying information included in web requests (e.g., making it so the destination server doesn't know who made the original request). Proxy servers can also be used to control internet use by employees and children (e.g., organizations and parents set up proxy servers to control and monitor how their employees or children use the internet) or to improve browsing speed and save bandwidth (e.g., a proxy server can cache popular websites (keeping a copy of the appropriate website data locally) and send the saved copy to the client when a request is made).

[0007] By allowing users to request content through a (remote) proxy server instead of accessing it directly, proxies can be used to bypass certain internet restrictions (such as firewalls). Proxy servers are often used to circumvent content restrictions based on geographic IP addresses. For example, if someone wants to obtain content from a US webpage but cannot access it from their home country, they can make the request through a proxy server located in the US (and with a US IP address). Using a proxy service, the user's traffic appears to originate from a US IP address. Proxy servers can also be used for web scraping, data mining, and other similar tasks.

[0008] Collecting data from the web is an important and often crucial task for many business sectors. As the importance and value of data continue to increase, the demand for more efficient data retrieval and analysis tools is also growing. The two main tools for accessing data sources on the web are web crawling and web scraping.

[0009] The general purpose of web crawling is to access publicly available web resources (usually web pages), retrieve their content, and forward the content to users. The fundamental goal of web crawling is to analyze the data retrieved by web crawlers and extract the information requested by users.

[0010] A web crawler (also known as a web spider or robot) is a program or automated script that browses the web in a structured and automated manner. Its primary task is to collect information requested by users. Web crawlers do this by browsing the web for the requested information and retrieving the raw Hypertext Markup Language (HTML) code of the websites they visit. Another purpose of web crawling is to index web content through search engines.

[0011] Web crawling (also known as web scraping or web data extraction) is the process of acquiring accessed web pages and parsing the content of requested data. Web crawling can be done manually, but it usually consists of automated processes implemented using web crawlers.

[0012] Web crawling and web scraping tools are often used together because scraping capabilities largely depend on the results of web crawling. More specifically, web crawling is the main component of web scraping. The data accessed and retrieved by the web crawler is then processed by the web scraper, which analyzes the retrieved content and extracts the desired data.

[0013] Web crawlers operate through a series of tasks, including accepting requests, collecting data, and delivering data. More specifically, a web crawler's tasks include: accepting requests from users; managing request queues; balancing the load across processing services; attempting to retrieve requested data; storing data; and delivering data. The data retrieval tasks also include managing Hypertext Transfer Protocol (HTTP) and Secure Hypertext Transfer Protocol (HTTPS) requests, rotating proxies, and headers.

[0014] Web crawlers use the standard HTTP / HTTPS request / response model to invoke and access targets. This model describes client-server communication via the web. The client sends an HTTP request to the server, requesting access to specific data. The server sends an HTTP response containing the requested data back to the client. The HTTP request indicates the target's Uniform Resource Locator (URL), access method, and headers. The HTTP response returns status information, headers, and the requested data.

[0015] HTTP headers enable clients and servers to pass additional information in HTTP requests or responses. Headers can be grouped according to their context.

[0016] General headers apply to both requests and responses, but are unrelated to the data transmitted in the body. Request headers contain more information about the resource to be retrieved or about the client requesting the resource. Response headers maintain additional information about the response, such as the URL of the source webpage or the server providing the response. Entity headers contain information about the body of the retrieved webpage, such as its content length or content type.

[0017] You can also group headers based on how the proxy handles them, such as Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, etc.

[0018] End-to-end headers: These headers must be transmitted to the final recipient of the message: the requesting server or the responding client. Intermediate proxies must retransmit these headers unmodified, and caches must store them.

[0019] Hop-by-hop headers: These headers are only meaningful for a single transport-level connection and must not be further retransmitted or cached by the agent. It should be noted that only hop-by-hop headers can be set using the Connection generic headers. An important detail regarding the functionality of hop-by-hop headers is that they are only transmitted between the requesting device and the agent, and their primary purpose is to support the management of the corresponding agent session.

[0020] An HTTP cookie (web cookie, browser cookie) is a small piece of data that a server sends to a user's web browser. Browsers can store cookies and send them back to the same server with subsequent requests. Typically, cookies are used to determine if two requests are from the same browser—for example, to keep a user logged in. Cookies store state information from the stateless HTTP protocol.

[0021] Cookies are primarily used for three purposes: 1) Session management (login, shopping cart, game scores, or anything else the server should remember); 2) Personalization (user preferences, themes, and other settings); 3) Tracking (recording and analyzing user behavior).

[0022] The Cookie header of an HTTP request contains stored HTTP cookies that were previously sent by the server using the Set-Cookie header. The Cookie header is optional and can be omitted if, for example, a browser's privacy settings block cookies.

[0023] The GET method is used to retrieve information from a target. When adhering to REST API guidelines, requests using the GET method only retrieve but do not modify or otherwise affect the source data. The POST method is used to submit data to a specified resource. Requests created using the POST method, or, if not strictly adhering to REST guidelines, update the source data.

[0024] The POST method allows users to send data to a web target and submit it. This data is included in the HTTP message body carried by the HTTP request. This data needs to be encoded before being sent through the system's operational steps. Data encoding is performed for data security reasons. Data encoding is also performed before the HTTP request is sent to a proxy server.

[0025] Protocols used to carry connection information across proxies are called proxy protocols. Proxy protocols provide a convenient way to securely transmit connection information, such as client addresses across multiple layers of Network Address Translation (NAT) or TCP proxies. They are designed to require minimal changes to existing components and limit the performance impact caused by the processing of the transmitted information.

[0026] The proxy header is a block of connection information added by the connection initiator at the start of the connection. From the protocol's perspective, this makes the header an authoritative designation of the connection source.

[0027] The goal of the proxy protocol is to populate the server's internal structure with information collected by the proxy—information the server would otherwise obtain itself if the client were connecting directly to the server instead of through the proxy. The information carried by the protocol is the kind of information the server would obtain using the getsockname() and getpeername() methods:

[0028] - Address families (AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX)

[0029] - Socket protocols (SOCK_STREAM for TCP, SOCK_DGRAM for UDP)

[0030] Layer 3 (IP) source and destination addresses

[0031] - Layer 4 (e.g., TCP or UDP) source port and destination port (if any)

[0032] When the socket buffer is still empty at the start of the connection, the entire header will always be delivered immediately. The sender must always ensure that the header is sent immediately so that the transport layer maintains atomicity along the path to the receiver. The receiver can tolerate a partial header or can simply discard the connection upon receiving a partial header.

[0033] Web crawlers have two main approaches to accomplish their tasks: synchronous and asynchronous. Synchronous methods (also known as real-time methods) use the same connection to fetch data for the user. Asynchronous methods (also known as callback methods) establish and use different connections for data delivery. The examples presented typically operate within the synchronous web crawling model.

[0034] When using synchronous data delivery methods, the user submits a request to the web crawler, and the crawler returns the requested data over the same open HTTP(S) connection. Because asynchronous data delivery methods offer the option to decide when to retrieve the requested data, there is no need to maintain an open connection between the client and the backend service (such as the web crawler). The web crawler notifies the user when the data is ready. In both cases, the web crawler's task is defined by providing a list of target URLs that the crawler must attempt to access. Summary of the Invention

[0035] Several issues exist related to web crawler performance, including insufficient information gathered from content sources, inadequate request characteristics, and limited capabilities of the proxies employed. Next-generation proxies can be used for additional purposes, rather than simply forwarding traffic from users and returning information. Next-generation proxies can examine user information and modify it if necessary for better crawling results. With next-generation proxies, proxy-dependent logic can continue to be used. Furthermore, next-generation proxies can perform most of the unblocking logic that users would otherwise need to manage themselves. Users do not need to know the logic for overcoming blocking restrictions; it is part of the crawling functionality adopted by the service provider. Next-generation proxies have the ability to adapt crawling logic to bypass potential limitations and obstacles.

[0036] In one aspect, the embodiments detailed herein disclose a method for optimizing parameters of a user's crawl request on the service provider side. When a crawl request for a target is processed via the HTTPS protocol, a standard Transport Layer Security (TLS) handshake occurs between the service provider's infrastructure (i.e., the actual crawling agent) and the target, while the egress nodes for the request are traversed using the HTTP method CONNECT, which is used to establish a network transport layer tunnel to the target on behalf of the crawling device.

[0037] The embodiments detailed herein extend the methods for task implementation by targeting a step in the web crawling process—the step where the client submits a request to the web crawler. The embodiments detailed herein allow for defining more complex requests for the web crawler and thus receiving more specific data. The data acquisition system and methods for the web crawler remain unchanged. This includes the same general web crawler task flow of accepting requests from users, attempting to retrieve the requested data, and delivering the data to the user.

[0038] The embodiments detailed herein operate within the environment of a data acquisition system (i.e., user-data acquisition system-target on the web) and elaborate on the inseparable structural and functional elements of the data acquisition system. Essentially, the embodiments detailed herein enhance the system in three ways: task definition, secure task implementation, and result optimization. The embodiments detailed herein allow users to define more comprehensive tasks for the data acquisition system, including the ability to submit HTTP request headers and cookies, rather than simply submitting URLs in a minimal data acquisition system lacking rich request parameter awareness.

[0039] Next-generation proxies allow users to create tokens that the proxy can use to reuse the same IP address or session. In the operation of a data acquisition system, sessions are maintained by keeping the same exit node of the proxy server for multiple HTTP requests.

[0040] In addition to the standard functionality of this feature, the embodiments detailed herein also allow sending custom HTTP headers and cookies through reserved egress nodes. Support for POST requests allows users to send data to the web target during the same session. In the normal operation of a data retrieval system, the GET method is used in an HTTP request to invoke the target. The embodiments detailed herein also allow the use of the POST method. While the GET method, which is the default setting, does not require declaring the HTTP request, the POST method requires explicit declaration.

[0041] The embodiments detailed herein allow users to declare specific (instead of the default) HTTP response status codes as transitive. The system treats a response with the declared status code as correct and passes the retrieved data to the next data processing step. The specific status code is declared in the HTTP request message.

[0042] The embodiments detailed in this article allow checking if a user's URL is correct, and if not, modifying the URL (in a way that the URL is not blocked by the target). The solution also allows modifying HTTP headers and cookies, or adding the necessary HTTP headers or cookies if they are not adequately configured or set by the user for use with the proxy.

[0043] The implementation detailed in this document allows for retries of user requests without notifying the user. The user does not need to repeatedly resend the request. The next-generation agent sees that the request was not fulfilled correctly and resends the request to the target again until a satisfactory result is obtained. Therefore, the next-generation agent significantly improves the success rate of crawling results.

[0044] Next-generation proxies can verify user-provided parameters in real time and directly notify users of any lack of technical details or inadequacies in the parameters to ensure an appropriate request-response session context. Next-generation proxies can also leverage the ability to crawl through a large pool of proxy IP addresses as an additional security layer against anti-scraping techniques that block or pollute results. Attached Figure Description

[0045] The features and advantages of the exemplary embodiments described herein will become apparent to those skilled in the art upon which this disclosure pertains after reading the following description with reference to the accompanying drawings, in which:

[0046] Figure 1 An exemplary component diagram is a context that displays the overall architecture of the component and at least one aspect of the solution's functionality.

[0047] Figure 2A This demonstrates an exemplary existing technology layout for the current application's crawling functionality.

[0048] Figure 2B An exemplary inventive implementation of a crawling technique with service provider-based crawling logic is demonstrated.

[0049] Figure 3A This is an exemplary sequence diagram describing the routing of crawl requests using a next-generation proxy.

[0050] Figure 3B This is an example diagram of a crawl request and is Figure 3A The continuation of.

[0051] Figure 4A This is an exemplary sequence diagram of a crawl request when the target returns an error response.

[0052] Figure 4B This is an exemplary sequence diagram of a crawl request when the target returns an error response, and it is... Figure 4A The continuation of.

[0053] Figure 4C This is an exemplary sequence diagram of a crawl request when the target returns an error response, and it is Figure 4A The continuation of.

[0054] Figure 5 This is an exemplary sequence diagram describing the next-generation agent's optimization of crawling requests.

[0055] Figure 6 It is an exemplary computing system that performs the disclosed methods. Detailed Implementation

[0056] Some general terms may be helpful and are included in this document for convenience, and are intended to be interpreted in the broadest possible sense.

[0057] Elements 104 to 110 identify portions of the service provider's infrastructure, while elements 102, 116, 118, and 120 show external components or systems.

[0058] User equipment 102 can be any suitable user computing device, including but not limited to smartphones, tablets, personal computing devices, laptops, gaming devices, vehicle infotainment systems, smart appliances (e.g., smart refrigerators or smart TVs), cloud servers, mainframes, laptops, desktop computers, workstations, mobile devices, or any other electronic device used to issue a grab request.

[0059] Request Optimizer 104 is a next-generation agent that is part of the service provider infrastructure that accepts requests from user devices and implements them through the system. Using analytics algorithms, the request optimizer can read the request's metadata and tailor the request using different pre-established crawling strategies to better target the desired outcome. It should be noted that response metadata, such as the URL and HTTP headers, is always visible to Request Optimizer 104. This functionality is essential for Request Optimizer 104 to properly process request parameters in order to analyze and tailor the request.

[0060] The Proxy Rotator 106 is part of the service provider's infrastructure and is coupled to separate external components that implement specific auxiliary functions. It is responsible for proxy control, rotation, maintenance, collection of statistics and reports.

[0061] Proxy A 108 and Proxy B 110 – These indicate exemplary proxy servers (computer systems or applications) opened for client connections, acting as intermediaries for requests from clients seeking resources from other servers. The client connects to the proxy server, requesting services such as files, links, web pages, or other resources available from different servers. The proxy server evaluates the requested content and forwards the request until it reaches the actual target resource or a resource containing the actual content. After obtaining the content, the proxy server typically forwards it to the original requester, but the proxy may also perform other actions (e.g., return an error message). In one aspect of at least one embodiment detailed herein, the proxy server may not have full visibility into the actual content obtained for the original requester; for example, in the case of an encrypted HTTPS session, if the proxy is not the decryption endpoint, the proxy, acting as an intermediary, blindly forwards data without knowing what it is forwarding. However, it should be noted that the metadata of the response is always visible to the service provider, such as HTTP headers. This functionality is necessary for the proxy to correctly forward the obtained data to the correct requester (end user or intermediary proxy device). Agent A and Agent B are presented here as a simple indication that more than one proxy server can be maintained at the service delivery infrastructure 114. This embodiment should not be limited to proxies belonging to the service provider. Agents can be owned and managed by any third party; however, the service provider always has access to and can use such proxies.

[0062] A proxy DB (112) is a structured database (DB) contained within a data storage device (e.g., a computer disk or computer memory). The specific details of the proxy DB implementation are independent of the functionality provided, whether it's a horizontally or vertically partitioned application; a hardware-based cluster; or an application-level distributed model. Any of these options operates within a single hardware device or in a distributed manner, and the data may be distributed across multiple schema instances, servers, data centers, countries, or continents for purposes such as load balancing, improved performance, ensuring platform resilience, or other objectives. The proxy DB is populated with data from proxy spinners and contains information about all proxies available to the service provider.

[0063] Target A 116 and Target B 118 are exemplary web servers serving content accessible via HTTP / HTTPS protocols. The two targets (Target A and Target B) are presented herein as a simple indication that more than one target may exist, but it should not be construed in any way as limiting the scope of this disclosure. An unlimited number of targets may exist on a network.

[0064] Network 120 - is a digital telecommunications network that allows nodes to share and access resources. Examples of networks: Local Area Network (LAN), Wide Area Network (WAN), Campus Area Network (CAN), Metropolitan Area Network (MAN), Home Area Network (HAN), Intranet, Extranet, Internet, Internet of Things.

[0065] Crawling logic 202 is the algorithm employed by request optimizer 104, whose specified task is to select various pre-established request parameters for a specific crawling query in order to search for and collect specific data from the target. Crawling logic 202 contains a large number of crawling strategies 504.

[0066] A crawling strategy 504 is a set of request parameters and their values ​​optimized for a specific type of request or target. The request optimizer 104 applies it to the original request based on decisions about what adjustments should be introduced. These decisions are based on crawling logic 202. Crawling strategy 504 can be a predefined artifact that is pre-designed, developed, and manually registered. Crawling strategy 504 can also be dynamically formed by crawling logic 202 based on automatic aggregation analysis of crawling results.

[0067] Positive responses are HTTP responses that are considered relevant and forwarded to the user by default, such as: 2xx - all successful responses; 4xx - client errors (with 5 exceptions: 429, 407, 408, 444, 499).

[0068] Negative responses are default system codes that users deem unacceptable HTTP responses or indicate unsuccessful results, such as: 429, 407, 408, 444, 499; 5xx - all server errors.

[0069] On one hand, the web scraping system detailed in this article uses the standard HTTP request / response model to invoke and access targets. This model describes client-server communication via the web. The client sends an HTTP request to the server, requesting access to specific data. The server sends an HTTP response containing the requested data back to the client. The HTTP request indicates the target URL, access method, headers, and request body. The HTTP response returns status information, headers, and the requested data.

[0070] In the normal operation of a data acquisition system, users can only control the URL parameters in the HTTP request. All other parameters of the request are controlled by the data acquisition system.

[0071] Figure 1An exemplary component diagram illustrating the overall architecture and at least one aspect of the solution functionality of the disclosed components is provided. In at least one aspect, this embodiment illustrates communication between user equipment 102 and service provider infrastructure 114 to reach any target 116, 118 on network 120. Service provider infrastructure 114 is composed of request optimizer 104, proxy rotator 106, proxy DB 112, and at least one proxy 108, 110. Figure 1 This indicates that user equipment 102 is not part of the service provider infrastructure 114, but rather exists as a separate device outside of that infrastructure. User equipment 102 can be any device that can connect to the service provider infrastructure via any network, but in most cases via the Internet. Such a device can be a computer, mobile phone, tablet computer, or any other electronic device. Figure 1 Targets 116 and 118 are located outside the service provider infrastructure 114 and are based on any network 120. In most cases, network 120 is the Internet. The request optimizer 104 allows users to modify request parameters to some extent—HTTP headers and cookies, HTTP methods (such as POST, GET, PUT, PATCH, and DELETE). Users also have the ability to establish HTTP sessions and define success conditions for requests to the target server.

[0072] Figure 2A This represents an exemplary prior art device in which the grabbing logic 202 is placed in the user device 102. Figure 2B The current embodiment is shown, where crawling logic 202 is located within service provider infrastructure 114. The crawling logic has hundreds or thousands of crawling strategies. Crawling strategies are developed by analyzing target parameters and defining best practices for crawling specific resources based on the response data of examining and evaluating the targets. At least some crawling request context parameters are established to select and apply appropriate crawling strategies:

[0073] - Establish an initial context for a crawl request that includes necessary parameters, such as headers presented to the web server during the initial contact, cookies, and user browser settings. Establishing this context may require a rigorous process, namely, pre-creating an organic, human-like browsing profile by visiting at least one page within the target web server that is different from the page intended for crawling, to populate the browsing profile with cookies from the target web server. Other activities for establishing an appropriate session context may include browsing multiple pages within the same target web server or presenting a history of visits to other web servers, activity on social networks, etc., to create a richer browsing profile.

[0074] - The appropriate geographical location for the last-mile agent.

[0075] - An appropriate timeout defined so that the page is considered to have been successfully rendered (if a headless browser is used as the application to fetch requests).

[0076] - Content elements that are required for a page to be considered successfully rendered (if a headless browser is used as the application to fetch requests).

[0077] - Guidelines for handling HTTP redirects.

[0078] - Define guidelines for which HTTP codes are considered signaling that should be retried when browsing attempts are blocked.

[0079] - Define guidelines for which HTTP codes are considered blocked by CAPTCHAs and retried accordingly.

[0080] - Define guidelines on which HTTP headers are mandatory for fetch requests, regardless of whether they are present in the original user request.

[0081] - Define guidelines for which HTTP cookies are mandatory for fetching requests, regardless of whether they are present in the original user request.

[0082] - Define guidelines for which HTTP headers must be replaced with their more effective counterparts (counterparts).

[0083] - Define guidelines for which HTTP headers must be removed, regardless of whether the user intends to include them in the original request.

[0084] - Define guidelines for which HTTP headers should appear in a fetch request (e.g., Chrome or Firefox's "User-Agent").

[0085] - Define the criteria for which type of proxy should be used for crawling requests (data center, residential, mobile, etc.). The main factor targeted by proxy selection is the target the strategy will be used for. Based on performing crawl tests and selecting options that provide the best success / failure rate results, create specialized strategies for specific targets. The application's crawling strategy can also depend on the content of the user's request: HTTP method, request parameters, user-defined specific requirements.

[0086] The embodiments detailed herein differ from current industry standards for managing crawling logic in several ways, including the fact that crawling functionality previously performed by users (i.e., customers) is now performed on the service provider side. In the prior art, service provider infrastructure 114 blindly forwards requests received from user equipment 102 to target 116 without modifying them. The new solution proposes a way to ensure more accurate responses from target 116 by having the service provider use crawling logic 202. The crawling logic comprises a set of pre-established rules (crawl strategies) that assign one or more specific crawling strategies to requests based on analysis of the requests. A pre-established set of request parameters and their values ​​optimized for specific types of requests or targets are placed in request optimizer 104, allowing request optimizer 104 to read and adjust queries received from user equipment 102.

[0087] Figure 3A It is a sequence diagram describing the routing of crawling requests. Figure 3B This is the follow-up process to the crawl request. The process begins once the user device sends a request to the service provider, more precisely, to the request optimizer 104 (step 302). The request optimizer 104 examines the request parameters (step 304), including at least the URL address, headers, and cookies. In certain cases, the request body can also be modified; for example, if the target of the crawl request is a webpage within a ".com" top-level domain (TLD), the policy will add a requirement to use a proxy with the United States as its geographic location. The HTTP response body is also analyzed to determine if the obtained data matches the request. Similarly, the parameters of the user's request can be changed to obtain better crawling results. An example of such a change can be described as follows:

[0088] A user dispatches a fetch request to a service provider, where the HTTP request body defines its content type as one of the Multipurpose Internet Mail Extensions (MIME) types for "compressed file," for example, in .zip format (MIME type = application / zip). However, the service provider identifies the target as a system that would work better with compressed data in .gz format (MIME type = application / gzip). The request optimization platform within the service provider's infrastructure examines the request headers and body, identifies this discrepancy, and adjusts them accordingly, converting the content to the appropriate format and updating the corresponding headers.

[0089] Another example of HTTP request body modification is when a user's request contains XML content in the body (i.e., MIME type = application / xml). The service provider has pre-established crawling logic guidelines that inform the target of the request that it is highly likely to block HTTP requests with human-unreadable XML content in the body. Therefore, the service provider targets the request to a feature of the web server, such as changing the content type (e.g., changing it to MIME type = text / xml). In this case, the request body and headers are modified by the service provider to avoid being blocked by the target and to receive better crawling results accordingly. In yet another example, the HTTP request body could be encrypted using the service provider's encryption toolset when the request is modified, and the artifacts describing the required information could be transmitted to the destination via an alternative channel or in a custom header within the request.

[0090] Therefore, if the parameters within the request are not considered to be aligned with the corresponding target in what is deemed the most efficient way, they are adjusted according to the fetching logic 202 loaded into the memory of the request optimizer 104 (step 306). The request optimizer 104 checks whether the URL address is correctly indicated, and if not, corrects the URL address in a manner that minimizes the blocking rate. The request optimizer 104 also checks the request headers and cookies and adjusts them in a manner that improves the quality of the request and the quality of the data expected to be received from the target.

[0091] In certain situations, the HTTP request body can also be modified. For example, if the target of the crawl request is in the ".com" TLD, the crawling logic guidelines stipulate that the request be geolocated to a US proxy to serve the request. If necessary, the HTTP request body can also be analyzed and adjusted to ensure better crawling results.

[0092] In at least one instance, after obtaining a request from the user and constructing the initial crawl session context, certain guidelines are applied to the user request before crawling the actual target page, for the specific strategy to be selected for the request:

[0093] - The browsing profile of a crawling proxy should have a certain level of detail, namely, whether the target's homepage should be visited before actual crawling, how many visits to the target's webpages are needed, whether only one visit to the homepage or other pages is needed to create an organic footprint of the user, and / or whether visits to webpages other than the target (such as social networks, entertainment sites, e-commerce stores, etc.) are needed.

[0094] - The geographic location of the agent provided by Agent Rotator 106.

[0095] - The timeout threshold used for rendering pages when using a headless browser.

[0096] - When using a headless browser, content elements must exist to be considered for rendering the page.

[0097] - Whether to follow redirection.

[0098] - This should be treated as an HTTP status code indicating that the device is blocked and should be retried accordingly.

[0099] - This should be treated as an HTTP status code that is blocked by a CAPTCHA and should be retried accordingly.

[0100] - HTTP request headers should always be inserted into the request (regardless of whether the user initially included them).

[0101] - HTTP request cookies should always be inserted into the request (regardless of whether the user initially included them).

[0102] - The HTTP request header must be replaced with its better counterpart.

[0103] - HTTP request headers that must be removed from the request (even if the user includes them in the request).

[0104] - The type of HTTP request header that should be present (e.g., "User-Agent" for Chrome or Firefox).

[0105] - The appropriate agent type (data center, residential, mobile, etc.) should be used.

[0106] The process of optimizing HTTP requests submitted by clients is transparent; that is, once initiated by a user, the request is analyzed and tailored according to the selected crawling strategy without any interaction from the initiating user.

[0107] If the request optimizer 104 notices inefficiencies or errors in the request headers and cookies, it adjusts the parameters according to a pre-established set of rules that assign one or more specific crawling strategies to the request (applying crawling logic 202). HTTP headers are components of the request message and are used to pass additional information to the server about the requested target. They are defined as key-value pairs in the HTTP message. A sample HTTP request message may include the following parameters: target address (e.g., http: / / scraping.com:60000), language requirement (e.g., en-US); request content / type (e.g., application / octet-stream), indication of custom headers, geographic location (e.g., US), and cookie index (e.g., SID=1234567890; SLK_TJJ=0987654321). If the request does not require modification, the request optimizer 104 does not perform corrections; however, the request optimizer 104 examines the content of the request.

[0108] In at least one aspect, the embodiments detailed herein further explain how the request optimizer 104 obtains a proxy from the proxy rotator 106. Once the request parameters are adjusted, the request optimizer 104 requests a specific proxy from the proxy rotator 106 (step 308). The proxy rotator 106 accesses the proxy DB 112 and selects a specific proxy suitable for fulfilling the user's request (step 310). The proxy rotator 106 provides the request optimizer 104 with information about the proxy (step 312). Figure 3B from Figure 3A The steps shown continue to illustrate the steps. Request optimizer 104 implements the request via proxy 108 provided by proxy rotator 106 (step 314). After obtaining the proxy, request optimizer 104 continues to implement the request received from the user equipment (step 314). Proxy 108 obtains data from the indicated target 116 (step 316) and forwards the data to request optimizer 104 (step 318). Proxy 108 obtains data from target 116 (step 316). The target can be any web resource on the network, and the network is typically the Internet. Once proxy 108 obtains the data, proxy 108 sends the data back to request optimizer 104 (step 318). Proxy 108 does not modify the received data, nor does it check the range or quality of the data. Proxy 108 simply transmits the data received from target 116 to request optimizer 104. Request optimizer 104 checks the metadata of the response (step 320). Proxy 108 neither modifies nor reads the information obtained from target 116, as it merely forwards the data. However, the request optimizer 104 examines the metadata of the response, that is, it examines the parameters of the reply received from the target (step 320), such as the HTTP response code.

[0109] When a server response is received, the HTTP message carries details about the response status. The response status indicates the outcome of the attempt to reach target 116. Response status codes are groups of possible outcomes:

[0110] -2xx- Successful response (positive response);

[0111] -3xx- redirection (positive response);

[0112] -4xx- Client error (with exceptions, a positive response will be given);

[0113] -5xx- Server Error (Negative Response).

[0114] In the operation of web crawling and scraping systems, an HTTP response that is considered positive is passed to the next data processing step.

[0115] In normal system operation, users do not have control over this setting. By default, only 2xx codes are considered transitive. When the request optimizer 104 is implemented in the system, users can declare specific, rather than default, status codes as transitive. The system will treat a response with the declared status code as correct and pass the acquired data to the next data processing step (322).

[0116] Declare a specific status code in the HTTP request header using the "x-successful-status-codes" header, whose value contains a list of success status codes: [500, 502, 42069]. If the user does not declare a specific status code, the request optimizer 104 operates with the default settings.

[0117] -1xx- Message Response indicates that the request has been received and understood. It is a temporary posting.

[0118] Meanwhile, request processing continues. It reminds the client to wait for the final response.

[0119] The -2xx- system will treat the response as OK.

[0120] -3xx- By default, the system follows a limited number of redirections. When the limit is exhausted, the system registers failed jobs.

[0121] The -4xx- system will treat the response as OK, but there are five exception codes: 429, 407, 408, 444, and 499.

[0122] -5xx- The system will retry a limited number of times. When the limit is exhausted, the system will register the failed job.

[0123] If the user chooses to use custom HTTP request headers to declare non-default behavior for handling specific HTTP response status codes, the request optimizer 104 will act as follows:

[0124] -1xx- system notifications to users specify status codes that are intermittent and do not support changing how they are handled.

[0125] -2xx- The system notifies the user that the specified status code always indicates a successful operation.

[0126] And it does not support changing how they are processed.

[0127] -3xx- The system informs the user that it follows the default behavior of redirection instructions from the server.

[0128] -4xx- The system notifies the user that the specified status code has been returned to the user, ignoring excluded codes (listed below). If the user expects, excluded codes can be treated as positive results.

[0129] -5xx- If the specified code is a complete 5xx HTTP response code, the system accepts user changes to the default behavior. If the response code in the reply exactly matches the configuration, data is returned to the user. Otherwise, the system follows the default behavior for retrying the request.

[0130] 4xx exceptions (with five exception codes: 429, 407, 408, 444, and 499) are treated as positive results, and the data requested for such cases is returned to the user. If the system receives one of the five exceptions, it repeats the request again until a positive response is received or the number of attempts is exhausted. However, users can specify custom handling for the exception codes and have the data for such responses forwarded to the user.

[0131] If the target's response HTTP headers contain an HTTP response code declared as positive, then the request optimizer 104 forwards the data to user device 102 (step 322). In this step, user device 102 receives the actual data (e.g., the body of the HTTP response) and metadata—headers, one of which contains the HTTP response status code, and a custom header carrying details about the fetch job status as part of the metadata.

[0132] Figure 4A , 4B4C and 4C are exemplary series of sequence diagrams describing the routing of a fetch request when the target returns an error response. These diagrams essentially show the same steps of requesting a proxy by request optimizer 104 and using that proxy to fulfill the user's request. The process begins once the user device sends a request to the service provider, more specifically to request optimizer 104 (step 402). Request optimizer 104 checks the request parameters (step 404) along with headers and cookies. If the parameters are not indicated as correct, they are adjusted according to fetch logic 202 loaded into the memory of request optimizer 104 (step 406). In at least one embodiment detailed herein, this disclosure further explains that request optimizer 104 obtains a proxy from proxy rotator 106. The request optimizer then requests a proxy (step 408). Proxy rotator 106 obtains a proxy from proxy DB 112 (step 410) and provides the proxy to request optimizer 104 (step 412). After obtaining the proxy, request optimizer 104 continues to fulfill the request received from the user device (step 414). Agent 108 obtains data from the indicated target 116 (step 416) and forwards the data to request optimizer 104 (step 418). Agent 108 does not modify the received data in any way, nor does it check the range or quality of the data. Agent 108 simply transmits the data received from target 116 to request optimizer 104. Request optimizer 104 checks the received metadata (step 420), and if the response status is negative (e.g., 4xx code exceptions (such as 429, 407, 408, 444, 499) or 5xx code), request optimizer 104 treats the response as a failure and does not send data to the user (step 422). In such cases, request optimizer 104 adjusts the request parameters (step 424) and repeats the adjusted request. The request can be adjusted by changing some parameters of the request according to the direction within the selected crawling strategy, trying to use another pre-established crawling strategy, or trying to crawl the target without making any changes to the strategy.

[0133] The crawling request can be tailored based on the analysis of the response to the target. For example, if the metadata in the response presents an HTTP response code 504, the request optimizer can apply a crawling strategy that introduces additional headers, while an HTTP response code 403 may require a strategy that enriches the request with organic, user-like behavioral markers (e.g., cookies from previous accesses to the target or other web resources).

[0134] All parts of the target's response and its content may be relevant to deciding how to appropriately adjust the overall crawling strategy for the request. Details about how the target returns the requested data may also be relevant.

[0135] Request optimizer 104 requests a proxy from proxy rotator 106 (step 426). Proxy rotator 106 obtains a proxy from proxy DB 112 (step 428) and provides information about the proxy to request optimizer 104 (step 430). Request optimizer 104 uses proxy 108 to fulfill the request (step 432). Proxy 108 obtains data from target 116 (step 434) and forwards the data to request optimizer 104 (step 436). Request optimizer 104 checks the metadata (step 438) and if the response is positive, sends the data to user device 102 (step 440). If the response is not positive, the request optimizer repeats the steps of adjusting request parameters and sending the request to the target using the selected proxy (i.e., repeating steps 406 to 438). These steps of checking the response from the target and repeating the request if the response is negative can be repeated until a positive response is received or until the number of negative responses reaches a certain indicated number (e.g., the request optimizer 104 can be configured to adjust the parameters of the request 25 times or more, and if the response is still negative within the limit of 25 times, the negative result is sent to the user device 102).

[0136] On the other hand, in at least one embodiment detailed herein, request optimizer 104 allows for the maintenance of sessions for multiple HTTP requests, a characteristic of data acquisition systems. Here, the same egress node is used to implement the session for multiple HTTP requests. In an exemplary operation of request optimizer 104, the session includes the use of the same IP proxy, the same headers, and the same cookies throughout the session. The exemplary operation also enables sending GET and POST requests (i.e., receiving and sending data) during the same session. Request optimizer 104 can keep the same session using the same proxy 108 open for several consecutive requests in the context of the same crawling session from a single user's device. Initially, the initial user's request indicates that a persistent session should be maintained for future requests by creating a string-formatted identifier for the session (e.g., "1234567890abcdef"). Upon receiving a request with a session ID identifier, request optimizer 104 is adequately instructed to register the session as persistent for that particular proxy and register the connection with the indicated session identifier. To continue using the same proxy for the next request, the user provides the previously established session identifier when submitting the crawling request. The proxy management will provide the same proxy and re-establish the connection with the target through the same proxy. However, it should be noted that the session identifier has an expiration threshold, and it is considered an unauthorized intrusion when a) a predefined period has elapsed, or b) the number of requests that could utilize the same session has been executed. After expiration, the next request containing the expired session identifier will result in the initiation of a new persistent session.

[0137] The system creates a session with an indicated session_id, pairs the session with a specific IP proxy and specified HTTP headers, and maintains the same exit node throughout the session. Additionally, the system sets the following parameters for the session: session_uuid, session_expires_at, and session_remaining. These parameters are system-defined, meaning the user has no control over them. The session remains open until the session expires or until an error occurs.

[0138] Figure 5 An exemplary sequence diagram describing the optimization of a crawling request is presented. Request optimizer 104 receives a request with unoptimized logic (step 502). Request optimizer 104 selects an appropriate crawling strategy based on the request requirements and objectives (step 504), wherein various crawling strategies are created and stored in request optimizer 104. Request optimizer 104 can use any of these strategies or combinations thereof to tailor the request for better crawling results. Examples of such strategies could be introducing or adjusting custom headers that identify the geographic location, time zone, or regional language to be used in the request. Once the request is tailored, it undergoes further processing (step 506).

[0139] Any of the embodiments described above herein can be rearranged and / or combined with other embodiments. Therefore, the concepts herein are not limited to any particular embodiment disclosed herein. Furthermore, embodiments may be entirely in hardware form or may include both hardware and software elements. Some embodiments may be implemented in software, including but not limited to firmware, resident software, microcode, etc. Figure 6 A computing system 600 is illustrated, wherein a computer-readable medium 606 can provide instructions for performing any of the methods disclosed herein.

[0140] Furthermore, embodiments may take the form of a computer program product, accessible from a computer-readable medium 606, which provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, the computer-readable medium 606 may be any means capable of tangibly storing a program for use by or in conjunction with an instruction execution system, apparatus, or device (including computer system 600).

[0141] Media 606 can be any tangible electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer-readable media 606 include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), NAND flash memory, read-only memory (ROM), hard disk, and optical disc. Some examples of optical discs include compact disc-read-only memory (CD-ROM), compact disc-read / write (CD-R / W), and digital versatile disc (DVD).

[0142] A computing system 600 suitable for storing and / or executing program code may include one or more processors 602 directly or indirectly coupled to memory 608 via a system bus 610. Memory 608 may include local memory used during the actual execution of the program code, mass storage devices, and cache memory providing temporary storage for at least some program code to reduce the number of times code is retrieved from mass storage devices during execution. Input / output or I / O devices 604 (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediate I / O controller. Network adapters may also be coupled to the system to enable the computing system 600 to be coupled to other data processing systems, such as via a host system interface 612, or via an intermediate private or public network to a remote printer or storage device. Modems, cable modems, and Ethernet cards are just a few types of network adapters currently available.

[0143] While several embodiments have been described, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the embodiments detailed herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this teaching. Benefits, advantages, solutions to problems, and any one or more elements that may make any benefit, advantage, or solution occur or become more apparent should not be construed as key, essential, or fundamental features or elements of any or all claims. The invention is defined only by the appended claims, including any modifications made during the pending period of this application and all equivalents of those claims.

[0144] This disclosure presents a system and method for extracting and collecting data from a web server, comprising: comparing parameters of a request received from a user's device with a list of crawling strategies; selecting a crawling strategy from the list of crawling strategies that corresponds to the parameters of the request; identifying differences between the request and the crawling strategy; adjusting the parameters of the request according to the crawling strategy to form an adjusted request; sending the adjusted request to a target through a proxy selected according to the crawling strategy; and receiving a response from the target.

[0145] Upon receiving a request, the service provider infrastructure autonomously adjusts the parameters requested from the user's device.

[0146] The method also includes examining and analyzing the metadata of the response received from the target, and forwarding the response to the user's device based on the analysis of the metadata.

[0147] If the response received from the target is negative, the method is repeated to form different adjusted requests that are subsequently sent to the target by adjusting the parameters of the request according to different crawling strategies from the list of crawling strategies. The method is repeated using an alternative crawling strategy selected from the list of crawling strategies until a positive response is obtained or until the maximum number of attempts is reached.

[0148] During adjustment, at least one parameter from the following components of the request is adjusted: URL address; headers; cookies; request body. If the request received from the user's device aligns with one of the crawling strategies from the list, the request parameters are not adjusted.

[0149] The proxy is selected from a proxy database by the service provider's infrastructure. The proxy is selected based on a crawling strategy and includes geographic location settings, proxy type, proxy availability, or any combination thereof. The proxy can remain available to multiple user device requests.

[0150] The method uses a request optimizer, and its operation settings include at least one of the following: fetch strategy, session ID number, server accepted as the request initiator, operation time range, scheduled maintenance job, connection session timeout, request processing log, or any combination thereof, and such settings are stored in a database on disk, a flat file on disk, a database in memory, or other storage media.

[0151] The proxy server reaches a web server that provides responses accessible via HTTP / HTTPS protocols and is attached to the network. This method can use a proxy server integrated into or separate from the service provider's infrastructure.

[0152] The embodiments disclose a system for extracting and collecting data from a network, comprising at least: a proxy rotator operable to provide multiple proxies over the network; and a request optimizer operable to compare parameters of a request received from a user's device with a list of crawling strategies, select a crawling strategy from the list of crawling strategies that corresponds to the parameters of the request, identify differences between the request and the crawling strategy, adjust the parameters of the request according to the crawling strategy to form an adjusted request, and send the adjusted request to a target through a proxy selected by the proxy rotator from a proxy database according to the crawling strategy.

[0153] This disclosure also proposes a non-transitory computer-readable medium for retrieving and collecting data from a web server. The non-transitory computer-readable medium includes instructions that, when executed by a computing device within a service provider infrastructure, cause the computing device to: compare parameters of a request received from a user's device with a list of crawling strategies; select a crawling strategy from the list of crawling strategies corresponding to the parameters of the request; identify differences between the request and the crawling strategy; adjust the parameters of the request according to the crawling strategy to form an adjusted request; send the adjusted request to a target via a proxy selected according to the crawling strategy; and receive a response from the target. Subsequently, metadata of the response received from the target is examined and analyzed, and the response is forwarded to the user's device based on the analysis of the metadata. If the response received from the target is negative, the operation of the medium is repeated to form different adjusted requests subsequently sent to the target by adjusting the parameters of the request according to different crawling strategies from the list of crawling strategies. The operation of the medium is repeated using an alternative crawling strategy selected from the list of crawling strategies until a positive response is obtained or until a maximum number of attempts is reached. The proxy can remain available for device requests from multiple users. Adjust at least one of the following parameters of the request: URL address; headers; cookies; request body. If the request received from the user's device aligns with one of the crawling strategies from the list, then the request parameters are not adjusted.

[0154] Furthermore, in this document, relational terms such as "first" and "second" may be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes, has, comprises, or contains a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Elements preceded by "comprises…a," "has…a," "includes…a," or "contains…a" do not exclude the presence of additional identical elements in processes, methods, articles, and / or apparatuses that include, have, comprise, and / or contain that element, unless otherwise expressly stated herein. Unless otherwise expressly stated herein, the terms "a" and "an" are defined as one or more. The terms “about,” “approximately,” or any other version thereof are defined as close to, as understood by one of ordinary skill in the art. A device or structure that is “configured” in a certain way is configured at least in that way, but may also be configured in a way not listed. Indications for elements may be used in either the singular or plural form, but this does not limit the scope of this disclosure and the same teachings may apply to multiple objects, even if the object is referred to in its singular form in the present application.

[0155] It should be understood that some embodiments describe the use of one or more general-purpose or special-purpose databases (such as a "proxy DB" or similar database) containing a collection of information organized to make it easily accessible, manageable, and updatable. Computer databases typically contain aggregations of data records or files, and in this context, databases typically store various information and statistics about proxies or egress nodes, and information about utilization thresholds of egress node providers. Such databases may also contain information about clients, executed requests, networks used, proxies used, the type of proxy requested, and similar data. Databases are structured to facilitate the storage, retrieval, modification, and deletion of data in conjunction with various data processing operations.

[0156] The embodiments detailed herein are provided to allow the reader to quickly determine the nature of this technical disclosure. It is understood that this submission is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, multiple features are combined in various embodiments for the purpose of simplifying this disclosure. This approach of the disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the detailed description, each claim being an independent, separately claimed subject matter.

Claims

1. A method for retrieving and collecting data from a web server, performed by a service provider infrastructure, comprising: The parameters of the request received from the user device are compared with the list of crawling strategies; Select the crawling strategy that corresponds to the parameters in the request from the list of crawling strategies; Identify the differences between the request and the crawling strategy; Adjust the parameters of the request according to the crawling strategy to form an adjusted request; The modified request is sent to the target by the agent selected according to the crawling strategy; Receive a response from the target; If the response received from the target is negative, the process of analyzing the metadata of the response, adjusting the parameters of the request according to different crawling strategies from the list of crawling strategies, and sending different adjusted requests to the target is repeated until a positive response is obtained or until the maximum number of attempts is reached.

2. The method according to claim 1, further comprising: Examine and analyze the metadata of the responses received from the target.

3. The method according to claim 2, further comprising: The response will be forwarded to the user's device based on the analysis of the metadata.

4. The method of claim 1, wherein at least one parameter among the following components of the adjustment request: URL address; header; cookie; Requesting subject.

5. The method of claim 1, wherein the agent is selected from an agent database by the service provider infrastructure.

6. The method of claim 5, wherein the agent is selected based on a crawling strategy and includes geographic location settings, agent type, agent availability, or any combination thereof.

7. The method of claim 1, wherein the agent remains available to multiple user equipment requests.

8. The method of claim 1, wherein the objective is to provide a web server capable of responding via HTTP / HTTPS protocols and attached to a network.

9. The method of claim 1, wherein the proxy is a proxy server integrated into the service provider's infrastructure.

10. The method of claim 1, wherein the proxy is a proxy server that is separate from the service provider infrastructure.

11. The method of claim 1, wherein upon receiving a request, the service provider infrastructure autonomously adjusts the parameters of the request from the user equipment.

12. A system for extracting and collecting data from a network, comprising at least: A proxy rotator, which is operable to provide multiple proxies over a network; as well as Request optimizer, the request optimizer being operable to: The parameters of the request received from the user device are compared with the list of crawling strategies. Select the crawling strategy that corresponds to the parameters in the request from the list of crawling strategies. Identify the differences between the request and the crawling strategy. Adjust the parameters of the request based on the crawling strategy to form a modified request. The modified request is sent to the target by an agent selected from the agent database by the agent rotator according to the crawling strategy, and Receive a response from the target; Furthermore, if the response received from the target is negative, the process of analyzing the metadata of the response, adjusting the parameters of the request according to different crawling strategies from the list of crawling strategies, and sending different adjusted requests to the target is repeated until a positive response is obtained or until the maximum number of attempts is reached.

13. The system of claim 12, wherein the request optimizer is capable of adjusting requests received from the user equipment by modifying at least one parameter among the following components: URL address; header; cookie; Requesting subject.

14. The system of claim 12, wherein the request optimizer uses the same agent for an unlimited number of requests received from the same user equipment.

15. The system of claim 12, wherein the request optimizer operation settings include at least one of the following: fetching strategy, session ID number, server accepted as request initiator, operation time range, scheduled maintenance job, connection session timeout, request processing log or any combination thereof, and such settings are stored in a database on disk, a flat file on disk, a database in memory or other storage medium.

16. A non-transitory computer-readable medium for retrieving and collecting data from a web server, the non-transitory computer-readable medium comprising instructions that, when executed by a computing device within a service provider infrastructure, cause the computing device to perform: The parameters of the request received from the user device are compared with the list of crawling strategies; Select the crawling strategy that corresponds to the parameters in the request from the list of crawling strategies; Identify the differences between the request and the crawling strategy; Adjust the parameters of the request according to the crawling strategy to form an adjusted request; The modified request is sent to the target by the agent selected according to the crawling strategy; Receive a response from the target; If the response received from the target is negative, the process of analyzing the metadata of the response, adjusting the parameters of the request according to different crawling strategies from the list of crawling strategies, and sending different adjusted requests to the target is repeated until a positive response is obtained or until the maximum number of attempts is reached.

17. The non-transitory computer-readable medium of claim 16, further comprising instructions that, when executed by the computing device, cause the computing device to perform the following operations: Examine and analyze the metadata of the responses received from the target.

18. The non-transitory computer-readable medium of claim 17, further comprising instructions that, when executed by the computing device, cause the computing device to perform the following operations: The response will be forwarded to the user's device based on the analysis of the metadata.

19. The non-transitory computer-readable medium of claim 16, wherein the agent remains available to multiple user equipment requests.

20. The non-transitory computer-readable medium of claim 16, wherein at least one of the following parameters is selected for the adjustment request: URL address; header; cookie; Requesting subject.

Images