A crawler implementation method and system for breaking through access restrictions of a server interface

Abstract

The application belongs to the technical field of web crawlers, and discloses a crawler implementation method and system for breaking through server interface access restrictions. A task pool of the system stores information such as each crawling target link and name, a crawler scheduling server is responsible for information data exchange between each module, a client is responsible for receiving task information and generating specific requests, a downloader is responsible for sending specific requests and returning response data, and a storage stores data into a database or a local text file. According to the application, a huge real device fingerprint and IP are used to solve the access restriction of the server interface of the crawler scheduling server, so that the crawler can stably and efficiently capture information data.

Description

Technical Field

[0001] This invention belongs to the technical field of web crawlers, and particularly relates to a method and system for implementing a web crawler that overcomes server interface access restrictions. Background Technology

[0002] A web crawler (also known as a web spider, web robot, or more commonly, web crawler in the FOAF community) is a program or script that automatically retrieves information from the World Wide Web according to certain rules.

[0003] Collecting user credit data online is an important means of credit rating; for example, transaction records scraped from the Alipay website can reflect a user's economic strength. However, there are also artificially created technical obstacles in collecting this information.

[0004] Some websites implement IP restrictions to prevent web crawlers from accessing information. For example, a single IP address may be limited to 100 accesses per minute. In this case, a web crawler scheduling server can only make 100 network requests per minute, and the target server will reject the 101st request.

[0005] Currently, the common solution to the above limitations is to increase the number of servers, thereby increasing the number of IP addresses. For example, if a single IP address is limited to 100 accesses per minute, increasing the number of IP addresses to 500 would allow for 50,000 requests per minute. While this solves the problem, it is extremely expensive and uneconomical.

[0006] Common methods used by web crawlers to bypass access restrictions include the following.

[0007] (1) Constructing a reasonable HTTP request header: Although websites may check every attribute of the HTTP request header, the most important parameter is usually the User-Agent. Regardless of the type of crawler, always remember to set the User-Agent attribute to something that is unlikely to arouse suspicion. For example, if you are using Python's urllib standard library, its User-Agent might be Python-urllib / 3.4, which is easily blocked. Additionally, if you are dealing with a highly vigilant website, pay attention to frequently used but rarely checked request headers, such as the Accept-Language attribute, which may be the key for the website to determine if you are a human visitor. Some websites also check the Referer; for example, some resource websites' anti-hotlinking measures detect the Referer. If you encounter this type of anti-crawler mechanism, you can directly add headers to the crawler, copying the browser's User-Agent into the crawler's headers; or modify the Referer value to the target website's domain name. For anti-crawler mechanisms that detect headers, modifying or adding headers in the crawler can effectively bypass them.

[0008] (2) Setting cookies: Cookies are encrypted data stored on the user's terminal. Some websites use cookies to identify users. If a user makes frequent requests, the website may notice and suspect the user of being a web crawler. In this case, the website can use the cookie to find the user and deny them access. To resolve this, you can disable cookies to actively prevent the server from writing to them. Disabling cookies can prevent websites that use cookies to identify web crawlers from blocking them. However, cookies are indispensable when scraping certain websites. To maintain a continuous login status on a website, a cookie needs to be stored across multiple pages. Some websites do not require a new cookie for each login; simply storing an existing login cookie is sufficient for access.

[0009] (3) Reduce access frequency: To improve crawler efficiency, many people use multithreading and asynchronous methods, allowing crawlers to access hundreds of thousands of pages in an hour, putting considerable pressure on the server. Therefore, some websites with robust security measures may block rapid form submissions or rapid interactions with the website. Even without these security measures, downloading large amounts of information from a website at a speed much faster than the average person may be blocked. For such websites, if crawler efficiency is not a primary concern, you can try reducing the crawler's access speed, such as ceasing to use multithreading, setting a waiting time after each access, or limiting the number of accesses per hour.

[0010] (4) Anti-crawling measures for dynamic pages: Most of the situations mentioned above occur on static pages. However, some websites require data to be crawled via AJAX requests or generated using JavaScript. In such cases, the network request needs to be analyzed first. If the AJAX request can be found, and the specific parameters and meaning of the response can be analyzed, the above method can be used to directly simulate the AJAX request and analyze the JSON response to obtain the required data. If this method fails, or if JavaScript processing is required, then Selenium + PhantomJS can be used. PhantomJS is a headless WebKit browser, essentially the same as a regular browser except for the lack of a user interface. Selenium is an automated testing tool that can drive the browser. Combining the two is a powerful combination, simulating everything from filling out forms to clicking buttons to scrolling through pages, without considering the specific request and response processes; it simply simulates the entire process of a person browsing a page and obtaining data. This framework can bypass almost all anti-scraping measures because it doesn't disguise itself as a browser to obtain data (the aforementioned method of adding headers is, to some extent, a browser disguise). PhantomJS itself is a browser without a user interface; it's just that the browser is controlled by a program, not a human. Selenium + PhantomJS can accomplish many things; cookies, JavaScript, and headers can all be handled automatically. For example, the currently popular sliding CAPTCHA can be solved using them.

[0011] (5) Using Proxy: While web crawlers forge almost all data, one thing they cannot fake is their IP address. Therefore, many websites have implemented rules to block IPs to prevent crawling—a simple, direct, and highly effective method. If an IP address is blocked, a proxy can be used. A proxy is a third party between the user and the website: the user sends a request to the proxy, which then forwards it to the server, thus identifying the proxy as the website being accessed. The server will then recognize this access as a proxy header. Using multiple proxies simultaneously reduces the access volume of a single IP. For example, http: / / www.ip181.com is a website that can detect proxy IPs; it can identify the IP being used, which can be used to verify whether a proxy IP has been successfully used. If multiple proxy IPs are available, an IP pool can be created, and each access can be randomly selected from the pool. If the access fails, the IP can be removed from the pool.

[0012] (6) Applying distributed crawlers: Usually, servers collect a large amount of information about client devices and behaviors, generate a unique client identifier through their own algorithm, and finally determine whether the client's access interface behavior is a crawler based on this identity ID and a large model, thereby restricting the acquisition of valid data. To avoid this situation, a large number of devices or other real clients can be relied upon to break through the total access limit.

[0013] Based on the above analysis, the problems and defects of the existing technology are as follows: the existing technology is very unstable when crawling large amounts of data with high concurrency and high quality, and has problems such as low system robustness, poor scalability, high code redundancy, and low availability. Summary of the Invention

[0014] To overcome the problems existing in related technologies, the present invention discloses a method and system for implementing a web crawler that breaks through server interface access restrictions.

[0015] The technical solution is as follows: A method for bypassing server interface access restrictions in web crawling. This method generates crawling tasks for each page in batches and distributes them to the client. It relies on the device fingerprint and IP provided by the client to bypass the access restrictions of the crawler scheduling server, maintaining stable, sustainable, and scalable data crawling. Specifically, it includes the following steps:

[0016] S1, the crawler scheduling server retrieves tasks from the task pool, checks the online status of the clients in turn, and sends the tasks to the clients in order;

[0017] S2, the client generates a complete request based on the task information and sends it to the downloader;

[0018] S3, the downloader processes the request and sets the response processing results, including maximum timeout processing, various encoding processing, and exception callback processing, and returns the results to the crawler scheduling server;

[0019] S4, the crawler scheduling server extracts and cleans data into the storage;

[0020] S5, the memory can store data in a database or export it as needed.

[0021] In step S1, the crawler scheduling server collects information about each task, including the target link, name, periodicity and one-time settings, maximum time for each request, maximum number of accesses, and whether to switch clients; it performs an MD5 digest algorithm on the above information to obtain a value, which is stored in the task pool, and the uniqueness of each task is guaranteed based on the set characteristics;

[0022] The crawler scheduling server checks the clients connected to it in turn, and distributes tasks to online clients in a round-robin manner to ensure that each client is evenly distributed with tasks.

[0023] In step S2, after obtaining the task, the client constructs a complete request body based on the task information. The task information includes user agent (UA), basic cookie information, device fingerprint, site IP information, and app identification information. The request body is then sent to the downloader and the task is marked as waiting.

[0024] In step S3, after the downloader receives the request body, it sets the request timeout and the maximum number of attempts, sets an exception callback method for each task to avoid abnormal termination of system operation, sends the request and returns data to the crawler scheduling server through a standardized interface, marks the task execution status according to the response status code, and returns the task to another queue in the task pool for the next trigger execution for tasks that have reached the maximum number of failures.

[0025] Whether a task is fixed on the current client is determined by the crawler scheduling server. If no fixed client is set when the task is created, the success rate of the task execution is based on the client.

[0026] Furthermore, exception callback methods are set, including:

[0027] Step 1: Construct two-dimensional callback models for different tasks, perform forward modeling to obtain normal running data, and construct task datasets based on the obtained normal running data;

[0028] Step 2: Construct a normally functioning inversion network and preprocess and partition the constructed task dataset;

[0029] The normally functioning inversion network has five convolutional kernels, three pooling layers, and two fully connected layers. The kernel size of the first convolutional layer is 1×11, the kernel size of the second convolutional layer is 1×5, and the kernel size of the other three convolutional layers is 1×3, with a stride of 1. The three pooling layers are all 1×3 with a stride of 2, and all use the VALID method. The preprocessing and partitioning of the constructed task dataset includes: using the first 80% of the dataset as the training set and the last 20% as the test set.

[0030] Step 3: Use the processed task dataset to train, optimize, and validate the constructed normal operation inversion network to obtain the trained normal operation inversion network;

[0031] First, set the parameters and train the AlexNet-Gra network using the divided training set; then, use the test set to verify if the AlexNet-Gra network achieves the expected results; if not, modify the parameters and train again; if the trained network achieves the expected results, input the validation set data and reconstruct the two-dimensional callback model; the parameter modifications include:

[0032] (1) Select an optimization function to optimize the weights and biases in the network that need to be updated; the expression of the optimization function is:

[0033]

[0034] In the formula, Let θ represent the weights and biases that need to be updated in the network, m represent the two-dimensional callback model, Θ represent the weights and biases that need to be updated in the network, L() represent the MSE loss function, and Net() represent the convolutional neural network representing the mapping from the observed data d to the predicted two-dimensional callback model, where d represents the observed data.

[0035] (2) Select the objective function, whose expression is:

[0036]

[0037] In the formula, L' is the objective function, N is the total number of data fitting iterations, i is the number of data fitting iterations for the i-th iteration, and d i Let m' be the number of observed data for the i-th data fitting, G be the M×N dimensional kernel matrix, and m' be the number of observed data. i For the two-dimensional callback model that fits the data in the i-th iteration, Let L(m,m') be the weight regularization value, L(m,m') be the model recovery degree, λ be the regularization parameter, and n be the size of the training set. For the degree of data fit, For weight decay regularization;

[0038] Step four: Input the normally functioning data into the trained normally functioning inversion network to obtain the inversion result data of abnormal task termination.

[0039] In step one, a two-dimensional callback model for different tasks is constructed, including:

[0040] The task space is divided into 800 rectangular cells of 20 rows × 40 columns, each cell being 25ms × 25ms in size. Multiple task nodes are set, with a transmission interval of 10ms between task nodes. Seven regular and unique models are constructed: 75ms × 75ms, 100ms × 100ms, 125ms × 125ms, 75ms × 150ms, 150ms × 75ms, 100ms × 200ms, and 200ms × 100ms.

[0041] In step one, forward modeling is performed to obtain normal operating data, including:

[0042] The mapping relationship from the two-dimensional callback model m to the normally running d is represented by matrix equations. Different callback values ​​are set for the model task to perform forward modeling and generate corresponding data.

[0043] The matrix equation is as follows:

[0044] d = Gm

[0045] d = (d1, d2, ..., d M ) T

[0046] m = (m1, m2, ..., m N ) T

[0047] Where T is the order, d M m is the observation data for the number of task nodes. N G is a two-dimensional callback model representing the number of grid cells, where G is an M×N dimensional kernel matrix, M is the number of task nodes, and N is the number of grid cells.

[0048] In step S4, after the crawler scheduling server receives the data returned by the downloader, it extracts, filters, and cleans the data before sending it to the storage. If the downloader returns abnormal data before the task reaches the maximum number of requests, it will choose whether to switch clients to continue executing the task based on the task settings information.

[0049] In step S5, the memory obtains the data sent by the crawler scheduling server, stores or updates the current data according to the APIs of various types of relational and non-relational databases, and supports online export of local text files based on task ID.

[0050] Another objective of this invention is to provide a web crawler implementation system that bypasses server interface access restrictions. This system implements the aforementioned web crawler implementation method for bypassing server interface access restrictions. The system includes:

[0051] The task pool is used by users to store crawling target information, including URL, task ID, site name, task execution status information, timeout, and maximum number of attempts.

[0052] The crawler scheduling server is used to issue a crawling task. The crawling task includes the task ID, the URL of the HTTP request and all parameters, and the maximum waiting time.

[0053] The client is used to immediately initiate an HTTP request to fetch the corresponding page after receiving the fetching task;

[0054] The downloader is used to send requests and handle various response results, including maximum timeout handling, various encoding processing, and exception callbacks.

[0055] The storage is used to extract, filter, clean, and persist the captured data into the database according to rules. It supports various relational and non-relational databases, as well as local text file export.

[0056] The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed;

[0057] The sending module is used to send the crawled data to the crawler scheduling server, while labeling it with the task ID. The crawled data is a string returned by the HTTP response.

[0058] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows: the system exponentially improves data acquisition efficiency, greatly reduces development, manpower, and equipment resource costs, and in today's information explosion era, the system is applicable to various large-scale data processing scenarios, such as large search engines, data mining analysis, and artificial intelligence. The technical solution of this invention has a low learning curve and high portability. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0060] Figure 1 This is a flowchart of a web crawler implementation method for overcoming server interface access restrictions provided in an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of a web crawler implementation system that overcomes server interface access restrictions, provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] The innovation of this invention lies in the fact that it integrates all stages of the crawler lifecycle, unifies the entry point, adopts a modular design concept, and fully improves the system's robustness and scalability, while also achieving an exponential increase in collection efficiency.

[0064] Example 1: The web crawler implementation method for overcoming server interface access restrictions provided in this embodiment of the invention includes:

[0065] S1, the crawler scheduling server retrieves tasks from the task pool, then checks the online status of the clients in turn and sends the tasks to the clients in order;

[0066] S2, the client generates a complete request based on the task information and sends it to the downloader, which includes common basic information about the device and site or app-specific identification information, while maintaining the integrity and authenticity of the access interface business;

[0067] S3, the downloader processes requests and sets various response results, including maximum timeout handling, various encoding processing, exception callbacks, etc., and finally returns the results to the crawler scheduling server.

[0068] S4, the crawler scheduling server extracts and cleans data into the storage according to rules;

[0069] S5, the memory can store data in a database or export it as needed.

[0070] In this embodiment of the invention, a crawler scheduling server is first implemented, which is responsible for communication and coordination between various modules of the entire system;

[0071] In step S1, information about each task is collected, including the target link, name, periodicity and one-time settings, maximum time for each request, maximum number of accesses, whether to switch clients, etc. Finally, the information is processed by the usual MD5 digest algorithm to obtain a value, which is stored in the task pool. Based on the set characteristics, the uniqueness of each task is guaranteed.

[0072] The crawler scheduling server sequentially checks the clients connected to it and distributes tasks to online clients in a round-robin manner, ensuring that tasks are distributed relatively evenly to each client and effectively and reasonably utilizing resources.

[0073] In step S2, after obtaining the task, the client constructs a complete request body based on the task information, which includes user agent (UA), basic cookie information, device fingerprint, site IP information, and app identification information. Then, it sends the request to the downloader and marks the task as waiting, while maintaining the integrity and authenticity of the access interface service.

[0074] In step S3, the downloader receives the request body, first sets the request timeout and the maximum number of attempts, then sets an exception callback method for each task to prevent any exception from terminating the system operation, and finally sends the request and returns data to the crawler scheduling server through a standardized interface. Of course, it will also mark the task execution status according to the response status code. For tasks that have reached the maximum number of failures, they will be returned to another queue in the task pool to wait for the next trigger for execution.

[0075] Whether a task is fixed to the current client is determined by the crawler scheduling server. Based on the success rate of the client's task execution, the client with a higher success rate will execute more tasks, provided that no fixed client is set when the task is created.

[0076] Furthermore, the exception callback method includes:

[0077] Step 1: Construct two-dimensional callback models for different tasks, perform forward modeling to obtain normal running data, and construct task datasets based on the obtained normal running data;

[0078] Step 2: Construct a normally functioning inversion network; and preprocess and partition the constructed task dataset.

[0079] Step 3: Use the processed task dataset to train, optimize, and validate the constructed normal operation inversion network to obtain the trained normal operation inversion network;

[0080] Step four: Input the normal running data into the trained normal running inversion network to obtain the inversion result data of abnormal task termination.

[0081] In step one, constructing two-dimensional callback models for different tasks includes:

[0082] The task space is divided into 800 rectangular cells, each with a size of 25ms × 25ms. Multiple task nodes are set, and the transmission interval between task nodes is 10ms. Seven regular and unique models are constructed: 3×3 (75ms × 75ms), 4×4 (100ms × 100ms), 5×5 (125ms × 125ms), 3×6 (75ms × 150ms), 6×3 (150ms × 75ms), 4×8 (100ms × 200ms), and 8×4 (200ms × 100ms).

[0083] In step one, obtaining normal operating data through forward modeling includes:

[0084] The mapping relationship from the two-dimensional callback model m to the normally running d can be represented by matrix equations; by setting different callback values ​​for the model task and performing forward modeling, the corresponding data can be generated.

[0085] The matrix equation is as follows:

[0086] d = Gm

[0087] d = (d1, d2, ..., d M ) T

[0088] m = (m1, m2, ..., m N )T

[0089] Where T is the order, d M m is the observation data for the number of task nodes. N G is a two-dimensional callback model representing the number of grid cells, where G is an M×N dimensional kernel matrix, M is the number of task nodes, and N is the number of grid cells.

[0090] In step two, the normally running inversion network has five convolutional kernels, three pooling layers, and two fully connected layers.

[0091] The kernel size of the first convolutional layer is 1×11, the kernel size of the second convolutional layer is 1×5, and the kernel size of the other convolutional layers is 1×3, with a stride of 1 for each layer.

[0092] The three pooling layers are all 1×3 in size and have a step size of 2; all are performed using the VALID method.

[0093] The division of the constructed task dataset includes: using the first 80% of the dataset as the training set and the last 20% as the test set;

[0094] The AlexNet-Gra network constructed by training, optimizing, and validating using the processed task dataset includes:

[0095] Set parameters and train the AlexNet-Gra network using the divided training set; then use the test set to verify whether the AlexNet-Gra network achieves the expected results; if not, modify the parameters and train again; if the trained network achieves the expected results, input the validation set data and reconstruct the two-dimensional callback model.

[0096] The modified parameters include:

[0097] (1) Select an optimization function to optimize the weights and biases in the network that need to be updated;

[0098] (2) Select the objective function:

[0099]

[0100] In the formula, L' is the objective function, N is the total number of data fitting iterations, i is the number of data fitting iterations for the i-th iteration, and d i Let m' be the number of observed data for the i-th data fitting, G be the M×N dimensional kernel matrix, and m' be the number of observed data. i For the two-dimensional callback model that fits the data in the i-th iteration, Let L(m,m') be the weight regularization value, L(m,m') be the model recovery degree, λ be the regularization parameter, and n be the size of the training set. For the degree of data fit, For weight decay regularization;

[0101] The optimization function is as follows:

[0102]

[0103] In the formula, Let θ represent the weights and biases that need to be updated in the network, m represent the two-dimensional callback model, Θ represent the weights and biases that need to be updated in the network, L() represent the MSE loss function, and Net() represent the convolutional neural network representing the mapping from the observed data d to the predicted two-dimensional callback model, where d represents the observed data.

[0104] For example, as another implementation, steps S1-S3 may also include the following judgment steps:

[0105] (1) The task ID is kept unique. It is based on the MD5 digest algorithm to calculate the task information as a whole, and based on the set characteristics, it ensures that there will be no duplicate IDs in the whole set.

[0106] (2) When the downloader sends a request, it will determine whether to retry the failed request based on the response status, and then query whether the task needs to switch clients to execute the current task based on the task ID.

[0107] (3) The crawler scheduling server will count the success rate of each client's task execution every half hour, remove the number of failures due to network, device and other factors, and roughly divide the clients into three categories (100-75, 75-50, less than 50). For those that need to switch clients to execute, they will be placed in the first echelon first, and if there are many tasks, they will be placed in the subsequent echelons in turn.

[0108] In step S4, after the crawler scheduling server receives the data returned by the downloader, it extracts, filters, and cleans the data according to the rules and sends it to the storage. If the data returned by the downloader is abnormal, it selects whether to switch clients to continue executing the task according to the task settings information, provided that the task has not reached the maximum number of requests.

[0109] In step S5, the memory obtains the data sent by the crawler scheduling server, and stores or updates the current data according to the APIs of various types of relational and non-relational databases. It also supports online export of local text files based on task ID. At this point, the entire life cycle ends.

[0110] Example 2, as Figure 2 As shown, as another embodiment of the present invention, the web crawler implementation method for overcoming server interface access restrictions provided by the embodiments of the present invention includes:

[0111] Batch generation of page crawling tasks is distributed to the client. The client provides a large number of device fingerprints and IPs to bypass the access restrictions of the crawler scheduling server, maintaining stable, sustainable and scalable data crawling.

[0112] The system takes the input site link, site name, category, and other important features, generates corresponding task IDs based on different time periods, and then sends this information to each client.

[0113] For each request, a maximum waiting time of 30 seconds is set. If the time limit is exceeded, the request is switched to a different client to prevent continuous resource occupation. Each task is executed at most three times. The task is issued in a round-robin manner, with the crawler scheduling server issuing tasks to each client in turn and marking the execution status of each task.

[0114] After the task of crawling the page is sent to the client, the client receives the task and immediately initiates the corresponding request to crawl the corresponding page. It supports various HTTP request methods (GET, POST, PUT, etc.). Each request has the client's real device fingerprint and IP, and maintains a long connection with the crawler scheduling server. From the server's perspective, it is considered that different users are accessing the page, thereby bypassing access restrictions.

[0115] The process of obtaining crawler information without exceeding the maximum waiting time and number of attempts includes: after the page is crawled, check if the maximum waiting time has been exceeded. If the maximum waiting time has not been exceeded, proceed to the next step; otherwise, switch clients to execute until the maximum number of attempts is reached. Finally, the crawler scheduling server issues the next crawling task step.

[0116] The crawler scheduling server standardizes the crawled data and returns it to the caller, while also marking the task execution status based on the response status.

[0117] To prevent tasks from being executed repeatedly and wasting resources, the crawler scheduling server, in principle, only issues each task once and stores its task ID in the cache. The corresponding ID is only cleared from the cache after the task is successfully executed and standardized data is obtained. For tasks that have reached the execution limit, the crawler scheduling server will store them in the failure queue, and the user can decide whether to re-execute or modify them.

[0118] After the data is obtained after batch execution of tasks, the crawler scheduling server will extract, clean and store various types of data. Data that does not meet the specifications will be filtered out and the task will be marked to wait for re-execution. It supports persistent storage in various relational and non-relational databases, and also supports exporting text files.

[0119] Example 3: This embodiment of the invention provides a web crawler implementation system that overcomes server interface access restrictions. This system is typically represented in the form of functional modules corresponding to the steps of the method. The system includes:

[0120] The task pool is responsible for storing the target information that users crawl, including the URL, task ID, site name, task execution status information, timeout, maximum number of attempts, etc.

[0121] The crawler scheduling server is used to issue a crawling task. The crawling task includes the task ID, the URL of the HTTP request and all parameters, and the maximum waiting time.

[0122] Specifically, it is responsible for core tasks such as task sending and receiving, data standardization, and client management. Its main task is to coordinate and manage the information transmission between various modules in a unified manner, and to provide a stable and efficient data capture and storage service.

[0123] The client is used to immediately initiate an HTTP request to fetch the corresponding page after receiving the fetching task;

[0124] Specifically, after receiving a crawling task, it is used to construct complete request information, including common basic information of the device and site or app-specific identification information, and to maintain the integrity and authenticity of the access interface business.

[0125] The downloader is responsible for sending requests and setting various response results, including handling maximum timeout, various encoding processes, and exception callbacks.

[0126] The storage is used to extract, filter, clean, and persist the captured data into the database according to rules. It supports various relational and non-relational databases and also supports exporting local text files (txt, excel, csv, etc.).

[0127] For example, the web crawler implementation system for overcoming server interface access restrictions provided in this embodiment of the invention includes:

[0128] The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed;

[0129] The sending module is used to send the crawled data to the crawler scheduling server, while labeling it with the task ID. The crawled data is a string returned by the HTTP response.

[0130] Preferably, the crawler implementation system that overcomes server interface access restrictions runs on a server, which provides a user input interface to implement the functions of the crawler implementation system that overcomes server interface access restrictions as described above when executed on an electronic device.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0134] This invention also provides a computer device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0135] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps described in the various method embodiments above.

[0136] This invention also provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, or switches.

[0137] This invention also provides a server that, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments.

[0138] This invention provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0140] To further illustrate the effects of the embodiments of the present invention, the following tests were conducted. A comparison was made between the top 100 most popular websites in China by frequently sending requests. The results showed that by directly accessing the websites, more than 500 requests were sent per hour, and 24% of the websites exhibited data acquisition anomalies. However, by using the technology of this patent, more than 500 requests were sent per hour, and only 3% of the websites showed data acquisition anomalies, reducing the probability of being detected by more than 87%.

[0141] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for implementing a web crawler that overcomes server interface access restrictions, characterized in that, This method generates crawling tasks for each page in batches and distributes them to clients. It relies on the device fingerprints and IPs provided by the clients to bypass the access restrictions of the crawler scheduling server, maintaining stable, sustainable, and scalable data crawling. Specifically, it includes the following steps: S1, the crawler scheduling server retrieves tasks from the task pool, checks the online status of the clients in turn, and sends the tasks to the clients in order; S2, the client generates a complete request based on the task information and sends it to the downloader; S3, the downloader processes the request and sets the response processing results, including maximum timeout processing, various encoding processing, and exception callback processing, and returns the results to the crawler scheduling server; S4, the crawler scheduling server extracts and cleans data into the storage; S5, the memory can store data into a database or export text as needed; In step S3, after the downloader receives the request body, it sets the request timeout and the maximum number of attempts, sets an exception callback method for each task to avoid abnormal termination of system operation, sends the request and returns data to the crawler scheduling server through a standardized interface, marks the task execution status according to the response status code, and returns the task to another queue in the task pool for the next trigger execution for tasks that have reached the maximum number of failures. Whether a task is fixed on the current client is determined by the crawler scheduling server. If no fixed client is set when the task is created, the success rate of the task execution by the client is used as the basis. Set exception callback methods, including: Step 1: Construct two-dimensional callback models for different tasks, perform forward modeling to obtain normal running data, and construct task datasets based on the obtained normal running data; Step 2: Construct a normally functioning inversion network and preprocess and partition the constructed task dataset; The normally functioning inversion network has five convolutional kernels, three pooling layers, and two fully connected layers. The first convolutional layer has a kernel size of 1×11, the second convolutional layer has a kernel size of 1×5, and the other three convolutional layers all have a kernel size of 1×3 with a stride of 1. The three pooling layers all have a kernel size of 1×3 with a stride of 2, and all use the VALID method. The preprocessing and partitioning of the constructed task dataset includes: using the first 80% of the dataset as the training set and the last 20% as the test set. Step 3: Use the processed task dataset to train, optimize, and validate the constructed normal operation inversion network to obtain the trained normal operation inversion network; First, set the parameters and train the AlexNet-Gra network using the divided training set; then, use the test set to verify if the AlexNet-Gra network achieves the expected results; if not, modify the parameters and train again; if the trained network achieves the expected results, input the validation set data and reconstruct the two-dimensional callback model; the parameter modifications include: (1) Select an optimization function to optimize the weights and biases in the network that need to be updated; the expression of the optimization function is: In the formula, For the optimized values ​​of weights and biases that need to be updated in the network, It is a two-dimensional callback model. For the weights and biases that need to be updated in the network, The MSE loss function is... For convolutional neural networks to represent data from observation data Mapping between the predictive two-dimensional callback model For observational data; (2) Select the objective function, the expression of which is: In the formula, Let be the objective function. This represents the total number of times the data was fitted. For the first Number of data fits For the first The number of observed data for the data fit. for dimensional kernel matrix, For the first A two-dimensional callback model for fitting the data to the desired number of data points. This is the weight regularization value. To the extent of model recovery, For regularization parameters, The size of the training set. For the degree of data fit, For weight decay regularization; Step four: Input the normally functioning data into the trained normally functioning inversion network to obtain the inversion result data of abnormal task termination.

2. The web crawler implementation method for overcoming server interface access restrictions according to claim 1, characterized in that, In step S1, the crawler scheduling server collects information about each task, including the target link, name, periodicity and one-time settings, maximum time for each request, maximum number of accesses, and whether to switch clients; it performs an MD5 digest algorithm on the above information to obtain a value, which is stored in the task pool, and the uniqueness of each task is guaranteed based on the set characteristics; The crawler scheduling server checks the clients connected to it in turn, and distributes tasks to online clients in a round-robin manner to ensure that each client is evenly distributed with tasks.

3. The web crawler implementation method for overcoming server interface access restrictions according to claim 1, characterized in that, In step S2, after obtaining the task, the client constructs a complete request body based on the task information. The task information includes user agent (UA), basic cookie information, device fingerprint, site IP information, and app identification information. The request body is then sent to the downloader and the task is marked as waiting.

4. The web crawler implementation method for overcoming server interface access restrictions according to claim 1, characterized in that, In step one, a two-dimensional callback model for different tasks is constructed, including: The task space is divided into 800 rectangular cells of 20 rows × 40 columns, each cell having a size of [size missing]. The task nodes are set to multiple, and the transmission interval between the task nodes is [missing information]. s, build , , , , , , There are a total of 7 models with regular and unique shapes.

5. The method for implementing a web crawler to bypass server interface access restrictions according to claim 1, characterized in that, In step one, forward modeling is performed to obtain normal operating data, including: Representing the two-dimensional callback model using matrix equations To run normally The mapping relationship is used to set different callback values ​​for the model task to perform forward modeling and generate corresponding data; The matrix equation is as follows: in, For order, For observational data on the number of task nodes, A two-dimensional callback model for the number of grid cells. for dimensional kernel matrix, This represents the number of task nodes. This represents the number of grid cells.

6. The web crawler implementation method for overcoming server interface access restrictions according to claim 1, characterized in that, In step S4, after the crawler scheduling server receives the data returned by the downloader, it extracts, filters, and cleans the data before sending it to the storage. If the downloader returns abnormal data before the task reaches the maximum number of requests, it will choose whether to switch clients to continue executing the task based on the task settings information.

7. The web crawler implementation method for overcoming server interface access restrictions according to claim 1, characterized in that, In step S5, the memory obtains the data sent by the crawler scheduling server, stores or updates the current data according to the APIs of various types of relational and non-relational databases, and supports online export of local text files based on task ID.

8. A web crawler implementation system that overcomes server interface access restrictions, characterized in that, This system implements the web crawler implementation method for bypassing server interface access restrictions as described in any one of claims 1-7, and the system includes: The task pool is used by users to store crawling target information, including URL, task ID, site name, task execution status information, timeout, and maximum number of attempts. The crawler scheduling server is used to issue a crawling task. The crawling task includes the task ID, the URL of the HTTP request and all parameters, and the maximum waiting time. The client is used to immediately initiate an HTTP request to fetch the corresponding page after receiving the fetching task; The downloader is used to send requests and handle various response results, including maximum timeout handling, various encoding processing, and exception callbacks. The storage is used to extract, filter, clean, and persist the captured data into the database according to rules. It supports various relational and non-relational databases, as well as local text file export. The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed; The sending module is used to send the crawled data to the crawler scheduling server, while labeling it with the task ID. The crawled data is a string returned by the HTTP response.

Images