Crawler implementation method and system capable of breaking through access limitation of server interface

Abstract

The invention belongs to the technical field of web crawlers, and discloses a crawler implementation method and system capable of breaking through server interface access limitation. A task pool of the system stores information such as links and names of crawling targets, a crawler scheduling server is responsible for information data exchange among modules, a client is responsible for receiving task information and generating specific requests, a downloader is responsible for sending the specific requests and returning response data, and a storage stores the data into a database or a local text file. According to huge real equipment fingerprints and IPs, the problem of interface access limitation of the crawler scheduling server is solved, so that the crawlers can stably and efficiently capture information data.

Description

Technical Field

[0001] The present invention belongs to the technical field of network crawlers, and in particular relates to a crawler implementation method and system that breaks through server interface access restrictions. Background Art

[0002] A web crawler (also known as a web spider, web robot, and more often a web chaser in the FOAF community) is a program or script that automatically crawls the World Wide Web according to certain rules.

[0003] Capturing users' credit data on the Internet is an important means of credit rating. For example, transaction records captured from the Alipay website can indirectly reflect the user's financial strength. However, when capturing this information, there are also artificial technical obstacles.

[0004] Some websites have IP restrictions to prevent crawlers from crawling information. For example, if a single IP is limited to 100 accesses per minute, then a crawler scheduling server can only initiate 100 network requests per minute, and when the 101st request is initiated, it will be rejected by the target server.

[0005] The common solution to the above limitations is to increase the number of servers and thus the number of IP addresses. For example, if a single IP is limited to 100 accesses per minute, then increasing the number of IPs to 500 can reach 50,000 requests per minute. Although this can solve the problem, it is very costly and uneconomical.

[0006] Common methods for web crawlers to break through access restrictions are as follows.

[0007] (1) Construct a reasonable HTTP request header: Although the website may check every attribute of the HTTP request header, the parameter that is really important is usually the User-Agent. No matter what kind of crawler you use, you must remember to set the User-Agent attribute to something that is not likely to arouse suspicion. For example, if you use the Python urllib standard library, its User-Agent may be Python-urllib / 3.4, which will easily be banned. In addition, if you are dealing with a very alert website, you should pay attention to those request headers that are often used but rarely checked, such as the Accept-Language attribute. Perhaps it is the key for that website to determine whether you are a human visitor. Some websites will also detect Referer. For example, the anti-hotlinking of some resource websites detects Referer. If you encounter this kind of anti-crawler mechanism, you can directly add Headers to the crawler and copy the browser's User-Agent to the crawler's Headers; or change the Referer value to the target website domain name. For anti-crawlers that detect Headers, modifying or adding Headers in the crawler can bypass them well.

[0008] (2) Setting cookies: Cookies are actually some encrypted data stored on the user's terminal. Some websites use cookies to identify users. If a certain visit always sends requests at a high frequency, it is likely to be noticed by the website and suspected of being a crawler. At this time, the website can find the user of this visit through cookies and deny access. In this case, you can disable cookies and actively prevent the server from writing. Disabling cookies can prevent websites that use cookies to identify crawlers from being banned. However, cookies are indispensable when collecting certain websites. To stay logged in on a website, you need to save a cookie on multiple pages. Some websites do not require a new cookie to be obtained every time you log in. You can access them by saving an old logged-in cookie.

[0009] (3) Reduce the access frequency: In order to improve the efficiency of crawlers, many people use multi-threading, asynchronous and other means in crawlers, so that crawlers can even visit hundreds of thousands of pages in an hour, which puts a lot of pressure on the server. Therefore, some websites with more complete protection measures may prevent quick submission of forms or quick interaction with the website. Even without these security measures, downloading a large amount of information from a website at a speed much faster than ordinary people may be blocked by the website. For such websites, if the crawler efficiency requirements are not high, you can try to reduce the crawler access speed, such as no longer using multi-threading technology, setting a waiting time after each visit, and limiting the number of visits per hour.

[0010] (4) Anti-crawling of dynamic pages: Most of the above situations occur on static pages. There are also some websites where the data to be crawled is obtained through ajax requests or generated by JavaScript. In this case, you first need to analyze the network request. If you can find the ajax request and analyze the specific parameters and the specific meaning of the response, you can use the above method to directly simulate the ajax request and analyze the response json to obtain the required data. If this method does not work or JS needs to be processed, then you can use selenium+phantomJS. PhantomJS is a webkit browser without an interface. It is basically the same as a general browser except that it has no interface. Selenium is an automated testing tool that can drive the browser. The combination of the two is a double-edged sword. From filling out forms to clicking buttons to scrolling pages, everything can be simulated. The specific request and response process is not considered. It just completely simulates the process of people browsing pages to obtain data. This framework can bypass most anti-crawlers, because it is not pretending to be a browser to obtain data (the above adding Headers is to a certain extent to pretend to be a browser). PhantomJS itself is a browser without an interface, but it is controlled by a program instead of a person. Selenium + phantomJS can do a lot of things, cookies, JavaScript, headers can be automatically processed, such as the popular sliding verification code, which can be solved with them.

[0011] (5) Use a proxy: Almost all data from web crawlers are fake, but there is one thing that cannot be faked, and that is the IP address. Therefore, in order to prevent crawlers, many websites have formulated a series of rules to block IPs, which is simple, crude and very effective. If the IP address is blocked, a proxy can be used. The so-called proxy is a third party between the user and the website: the user first sends the request to the proxy, and then the proxy sends it to the server, and it is considered that the proxy is accessing that website. At this time, the server will regard this access as a proxy header. When multiple proxies are used at the same time, the number of visits to a single IP will decrease. For example, http: / / www.ip181.com is a website that can detect proxy IPs. It can detect what IP is used, which can be used to check whether you have successfully used the proxy IP. If there are multiple proxy IPs, then you can establish an IP pool and randomly select one from the IP pool for access each time. If the access fails, remove it from the pool.

[0012] (6) Apply distributed crawlers: Usually the server will collect a large amount of information about the client's device, behavior, etc., generate a unique client identity through its own algorithm, and finally use this identity ID to train a large model to determine whether the client's access interface behavior is a crawler, thereby limiting its ability to obtain valid data. To avoid this situation, you can rely on a large number of devices or other real clients to break through the total access limit.

[0013] Through the above analysis, the problems and defects of the existing technology are as follows: the existing technology is very unstable when realizing large-scale, high-concurrency, and high-quality crawling data, and has problems such as low system robustness, poor scalability, high code redundancy, and low availability. Summary of the invention

[0014] In order to overcome the problems existing in the related art, the disclosed embodiment of the present invention provides a crawler implementation method and system that breaks through the server interface access restriction.

[0015] The technical solution is as follows: a crawler implementation method that breaks through the server interface access restrictions, which batch generates page crawling tasks and distributes them to the client, relies on the device fingerprint and IP provided by the client to break through the crawler scheduling server access restrictions, and maintains stable, sustainable, and scalable data crawling, specifically including the following steps:

[0016] S1, the crawler scheduling server takes tasks from the task pool, checks the online status of the client in sequence, and sends tasks to the client in sequence;

[0017] S2, the client generates a complete request based on the task information and gives it to the downloader;

[0018] S3, the downloader processes the request and sets the response processing results including maximum timeout processing, various encoding processing, and exception callback processing, and returns the results to the crawler scheduling server;

[0019] S4, the crawler dispatches the server to extract and clean the data into the storage;

[0020] S5, the memory is stored in a database or exported as a text according to needs.

[0021] In step S1, the crawler scheduling server collects information about each task, including target link, name, periodicity and one-time settings, maximum time for each request, maximum number of visits, and whether to switch clients; the above information is subjected to the md5 digest algorithm to obtain a value, which is stored in the task pool. Based on the set characteristics, the uniqueness of each task is guaranteed;

[0022] The crawler scheduling server starts to check the clients connected to the crawler scheduling server in turn, and issues tasks to the online clients in a round-robin manner to ensure that tasks are evenly distributed to each client.

[0023] In step S2, after obtaining the task, the client constructs a complete request body based on the task information; the task information includes basic information such as UA and cookies, device fingerprint, site IP information, and app identification information; sends it to the downloader and marks the task as being in a waiting state.

[0024] In step S3, after the downloader gets the request body, it sets the request timeout and the maximum number of attempts, sets an exception callback method for each task to avoid abnormal termination of system operation, sends the request and returns data to the crawler scheduling server through a standardized interface, marks the task execution status according to the response status code, and returns the task to another queue in the task pool for the task with the maximum number of failures, waiting for the next trigger execution;

[0025] Among them, whether a task is fixed on the current client is determined by the crawler scheduling server. If no fixed client is set for execution when the task is created, the success rate of the client executing the task is determined.

[0026] Furthermore, set the exception callback method, including:

[0027] Step 1: construct a two-dimensional callback model for different tasks, perform forward modeling to obtain normal operation data, and construct a task data set based on the obtained normal operation data;

[0028] Step 2: construct an inversion network that runs normally, and preprocess and divide the constructed task data set;

[0029] The normal inversion network has five convolution kernels, three pooling layers, and two fully connected layers; the convolution kernel size of the first convolution layer is 1×11, the convolution kernel size of the second convolution layer is 1×5, and the convolution kernel sizes of the other three convolution layers are all 1×3, and the step size is 1; the size of the three pooling layers is 1×3, and the step size is 2, all of which are performed in the VALID mode; the preprocessing and division of the constructed task data set includes: taking the first 80% of the data set as a training set and the last 20% as a test set;

[0030] Step 3, using the processed task data set to train, optimize, and verify the constructed normal operation inversion network to obtain a trained normal operation inversion network;

[0031] First, set the parameters and use the divided training set to train the AlexNet-Gra network; use the test set to verify whether the AlexNet-Gra network achieves the expected effect; if not, modify the parameters and train again; if the trained network achieves the expected effect, input the verification set data and reconstruct the two-dimensional callback model; the modified parameters include:

[0032] (1) Select an optimization function to optimize the weights and biases that need to be updated in the network; the expression of the optimization function is:

[0033]

[0034] In the formula, are the weights and bias optimization values ​​that need to be updated in the network, m is the two-dimensional callback model, Θ is the weights and biases that need to be updated in the network, L() is the MSE loss function, Net() is the convolutional neural network that represents the mapping from the observed data d to the predicted two-dimensional callback model, and d is the observed data;

[0035] (2) Select the objective function, the expression is:

[0036]

[0037] In the formula, L' is the objective function, N is the total number of data fitting, i is the number of data fitting, d i is the observed data of the i-th data fitting number, G is the M×N dimensional kernel matrix, m' i is the two-dimensional callback model of the fitting of the i-th data fitting number, is the weight regularization value, L(m,m') is the degree of model recovery, λ is the regularization parameter, n is the size of the training set, is the degree of data fitting, is weight decay regularization;

[0038] Step 4: Input the normal operation data into the trained normal operation inversion network to obtain the inverted task abnormal termination result data.

[0039] In step 1, a two-dimensional callback model for different tasks is constructed, including:

[0040] The task space is divided into 800 rectangular cells with 20 rows × 40 columns. The size of each cell is 25ms × 25ms. The task nodes are set to multiple, and the transmission interval between task nodes is 10ms. Seven regular and single models are constructed, namely 75ms × 75ms, 100ms × 100ms, 125ms × 125ms, 75ms × 150ms, 150ms × 75ms, 100ms × 200ms, and 200ms × 100ms.

[0041] In step 1, forward modeling is performed to obtain normal operation data, including:

[0042] The matrix equation is used to represent the mapping relationship between the two-dimensional callback model m and the normal operation d, and different callback values ​​are set for the model task to perform forward modeling and generate corresponding data;

[0043] The matrix equation is as follows:

[0044] d=Gm

[0045] d=(d1,d2,……,d M ) T

[0046] m=(m1,m2,……,m N ) T

[0047] Where T is the order, d M is the observed data of the number of task nodes, m N is a two-dimensional callback model with the number of grid units, G is an M×N dimensional kernel matrix, M is the number of task nodes, and N is the number of grid units.

[0048] In step S4, after obtaining the data returned by the downloader, the crawler scheduling server extracts, filters, cleans the data, and sends it to the storage. If the task has not reached the maximum number of requests and the data returned by the downloader is abnormal, it is determined whether to switch the client to continue the task based on the task setting information.

[0049] In step S5, the memory obtains the data sent by the crawler scheduling server, stores or updates the current data according to various types of relational and non-relational database APIs, and supports online export of local text files according to the task ID.

[0050] Another object of the present invention is to provide a crawler implementation system that breaks through the server interface access restriction, the system implements the crawler implementation method that breaks through the server interface access restriction, and the system includes:

[0051] The task pool is used for users to store crawling target information, including URL, task ID, site name, task execution status information, timeout, and maximum number of attempts;

[0052] The crawler scheduling server is used to issue a crawling task, which includes the task ID, the URL of the HTTP request, all parameters, and the maximum waiting time;

[0053] The client is used to initiate an HTTP request to crawl the corresponding page immediately after receiving the crawling task;

[0054] Downloader, used to send requests and set various response results, including maximum timeout processing, various encoding processing, and exception callback;

[0055] Storage, which is used to extract, filter, clean and persist the captured data into the database according to rules. It supports various relational and non-relational databases, as well as local text file export.

[0056] The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed;

[0057] The sending module is used to send the captured data to the crawler scheduling server and mark the task ID. The captured data is the string returned by the HTTP response.

[0058] Combining all the above technical solutions, the advantages and positive effects of the present invention are: the system exponentially improves data collection efficiency, greatly reduces the cost of development, manpower, equipment resources, etc. In addition, in today's information explosion era, the system can be applied to various large-scale data processing scenarios, such as large search engines, data mining analysis, artificial intelligence, etc. The technical solution of the present invention has a low learning threshold and high portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0060] Figure 1 It is a flow chart of a crawler implementation method for breaking through the server interface access restriction provided by an embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of a crawler implementation system for breaking through the server interface access restriction provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0063] The innovation of the present invention is that it integrates all stages of the crawler life cycle, unifies the entrance, adopts a modular design concept, fully improves the system robustness and scalability, and the collection efficiency also increases exponentially.

[0064] Embodiment 1, the crawler implementation method for breaking through the server interface access restriction provided by the embodiment of the present invention includes:

[0065] S1, the crawler scheduling server takes tasks from the task pool, then checks the online status of the client in turn and sends tasks to the client in sequence;

[0066] S2: The client generates a complete request based on the task information and sends it to the downloader, which contains common basic information of the device and identification information specific to the site or app, and maintains the integrity and authenticity of the access interface service;

[0067] S3, the downloader processes the request and sets various response results, including maximum timeout processing, various encoding processing, exception callback, etc., and finally returns the result to the crawler scheduling server;

[0068] S4, the crawler dispatches the server to extract and clean the data into the storage according to the rules;

[0069] S5, the memory is stored in a database or exported as a text according to needs.

[0070] In the embodiment of the present invention, a crawler scheduling server is first implemented to be responsible for the communication and coordination between the various modules of the entire system;

[0071] In step S1, collect each task information, including target link, name, periodicity and one-time settings, maximum time for each request, maximum number of accesses, whether to switch clients, etc. Finally, perform the usual md5 digest algorithm on this information as a whole to obtain a value, which is stored in the task pool. Based on the set characteristics, the uniqueness of each task is guaranteed;

[0072] The crawler scheduling server starts to check the clients connected to the crawler scheduling server in turn, and issues tasks to the online clients in a polling manner, ensuring that tasks are evenly distributed to each client and resources are occupied effectively and reasonably.

[0073] In step S2, after obtaining the task, the client constructs a complete request body based on the information of each task, which includes basic information such as UA and cookies, device fingerprint, site IP information, and app identification information; then sends it to the downloader and marks the task in a waiting state, and maintains the integrity and authenticity of the access interface service.

[0074] In step S3, the downloader gets the request body, first sets the request timeout and the maximum number of attempts, then sets the exception callback method for each task to prevent any abnormal termination of the system operation, and finally sends the request and returns data to the crawler scheduling server through the standardized interface. Of course, the task execution status will be marked according to the response status code. For tasks that have reached the maximum number of failures, they will be returned to another queue in the task pool and wait for the next trigger execution;

[0075] Whether a task is fixed on the current client will be determined by the crawler scheduling server. According to the success rate of the client executing the task, the client with a higher success rate will execute more tasks, provided that no fixed client execution is set when the task is created.

[0076] Furthermore, the abnormal callback method includes:

[0077] Step 1: construct a two-dimensional callback model for different tasks, perform forward modeling to obtain normal operation data, and construct a task data set based on the obtained normal operation data;

[0078] Step 2: construct an inversion network that operates normally; and preprocess and divide the constructed task data set;

[0079] Step 3, using the processed task data set to train, optimize, and verify the constructed normal operation inversion network to obtain a trained normal operation inversion network;

[0080] Step 4: Input the normal operation data into the trained normal operation inversion network to obtain the inverted task abnormal termination result data.

[0081] In step 1, the construction of two-dimensional callback models for different tasks includes:

[0082] The task space is divided into 800 rectangular cells of 20 rows × 40 columns, each cell is 25ms × 25ms in size, and the number of task nodes is multiple. The transmission interval between task nodes is 10ms. Seven regular and single models are constructed, namely 3×3 (75ms × 75ms), 4×4 (100ms × 100ms), 5×5 (125ms × 125ms), 3×6 (75ms × 150ms), 6×3 (150ms × 75ms), 4×8 (100ms × 200ms), and 8×4 (200ms × 100ms).

[0083] In step 1, the forward modeling to obtain normal operation data includes:

[0084] The mapping relationship between the two-dimensional callback model m and the normal operation d is expressed by using a matrix equation; different callback values ​​are set for the model task to generate corresponding data by forward modeling;

[0085] The matrix equation is as follows:

[0086] d=Gm

[0087] d=(d1,d2,……,d M ) T

[0088] m=(m1,m2,……,m N )T

[0089] Where T is the order, d M is the observed data of the number of task nodes, m N is a two-dimensional callback model with the number of grid units, G is an M×N dimensional kernel matrix, M is the number of task nodes, and N is the number of grid units.

[0090] In step 2, the normal inversion network has five convolution kernels, three pooling layers, and two fully connected layers;

[0091] The convolution kernel size of the first convolution layer is 1×11, the convolution kernel size of the second convolution layer is 1×5, and the convolution kernel size of other convolution layers is 1×3, and the step size is 1;

[0092] The sizes of the three pooling layers are all 1×3, and the step sizes are all 2; all are performed in the VALID mode;

[0093] The partitioning of the constructed task data set includes: using the first 80% of the data set as a training set and the last 20% as a test set;

[0094] The AlexNet-Gra network constructed by training, optimizing, and verifying the processed task data set includes:

[0095] Set parameters and use the divided training set to train the AlexNet-Gra network; and use the test set to verify whether the AlexNet-Gra network achieves the expected effect; if not, modify the parameters and train again; if the trained network achieves the expected effect, input the verification set data and reconstruct the two-dimensional callback model;

[0096] The modification parameters include:

[0097] (1) Select an optimization function to optimize the weights and biases that need to be updated in the network;

[0098] (2) Select the objective function:

[0099]

[0100] In the formula, L' is the objective function, N is the total number of data fitting, i is the number of data fitting, d i is the observed data of the i-th data fitting number, G is the M×N dimensional kernel matrix, m' i is the two-dimensional callback model of the fitting of the i-th data fitting number, is the weight regularization value, L(m,m') is the degree of model recovery, λ is the regularization parameter, n is the size of the training set, is the degree of data fitting, is weight decay regularization;

[0101] The optimization function is as follows:

[0102]

[0103] In the formula, are the weights and bias optimization values ​​that need to be updated in the network, m is the two-dimensional callback model, Θ is the weights and biases that need to be updated in the network, L() is the MSE loss function, Net() is the convolutional neural network that represents the mapping from the observed data d to the predicted two-dimensional callback model, and d is the observed data;

[0104] Exemplarily, as another implementation, the judgment steps that can also be adopted in steps S1-S3 include:

[0105] (1) The task ID remains unique. It calculates the task information as a whole based on the MD5 digest algorithm, and then based on the set characteristics, it ensures that there will be no duplicate IDs in the entire set.

[0106] (2) When sending a request, the downloader will determine whether to retry a failed request based on the response status, and then query whether the task needs to switch the client to execute the current task based on the task ID.

[0107] (3) The crawler scheduling server will count the success rate of each client's task execution every half an hour, excluding the number of failures due to factors such as network and equipment. The clients are roughly divided into three categories (100-75, 75-50, and less than 50). For those who need to switch clients for execution, they will be placed in the first echelon first. If there are more tasks, they will be placed in the subsequent echelons in turn.

[0108] In step S4, after obtaining the data returned by the downloader, the crawler scheduling server extracts, filters, and cleans the data according to the rules and sends it to the storage. If the data returned by the downloader is abnormal, it chooses whether to switch the client to continue executing the task according to the task setting information, provided that the task has not reached the maximum number of requests;

[0109] In step S5, the memory obtains the data sent by the crawler scheduling server, and stores or updates the current data according to various types of relational and non-relational database APIs. Of course, it also supports online export of local text files based on the task ID. The entire life cycle ends here.

[0110] Embodiment 2, as Figure 2 As shown, as another implementation mode of the present invention, the crawler implementation method for breaking through the server interface access restriction provided by the embodiment of the present invention includes:

[0111] Batch generate various page crawling tasks and distribute them to the client, relying on the client to provide a large number of device fingerprints and IP to break through the access restrictions of the crawler scheduling server and maintain stable, sustainable and scalable data crawling.

[0112] The important features such as the input site link, site name, category, etc. are used to generate corresponding task IDs according to different time periods, and finally this information is sent to each client.

[0113] The maximum waiting time for each request is set to 30 seconds. If the time exceeds the limit, different clients will be switched to execute to prevent resources from being occupied all the time. Each task is executed at most three times, and each task is issued based on the polling method. The crawler scheduling server sends tasks to each client in turn and marks the execution status of each task.

[0114] After the task of crawling the page is sent to the client, the client will immediately initiate a corresponding request to crawl the corresponding page after receiving the crawling task. It supports various http request methods (GET, POST, PUT, etc.). Each request has the client's real device fingerprint and IP, and maintains a long connection with the crawler scheduling server. For the server, it is considered that different real users are accessing the page, thereby breaking through the access restrictions.

[0115] The method of obtaining crawler crawling information without exceeding the maximum waiting time and the number of attempts includes: after the page crawling is completed, checking whether the maximum waiting time has been exceeded, if the maximum waiting time has not been exceeded, executing the next step, otherwise switching the client to execute until the maximum number of attempts has been reached, and finally the crawler scheduling server sends the next crawling task step;

[0116] The crawler scheduling server normalizes the captured data and returns it to the caller, while marking the task execution status based on the response status.

[0117] In order to prevent repeated execution of tasks and waste of resources, the crawler scheduling server only sends each task once in principle and stores its task ID in the cache. The corresponding ID is cleared from the cache only after the task is successfully executed and the standardized data is obtained. For tasks that have reached the upper limit of execution times, the crawler scheduling server will store them in the failed queue, and the user will decide whether to re-execute or modify them.

[0118] After the data is obtained after batch execution of tasks, the crawler scheduling server will extract, clean, and store various types of data, filter out data that does not meet the specifications, and mark the task waiting to be re-executed. It supports persistent storage in various relational and non-relational databases, and also supports text file export.

[0119] Embodiment 3, the embodiment of the present invention provides a crawler implementation system that breaks through the server interface access restriction, the system is usually represented in the form of functional modules corresponding to each step of the method. The system includes:

[0120] The task pool is responsible for storing crawling target information for users, including URL, task ID, site name, task execution status information, timeout, maximum number of attempts, etc.

[0121] The crawler scheduling server is used to issue a crawling task, which includes the task ID, the URL of the HTTP request, all parameters, and the maximum waiting time;

[0122] Specifically, it is responsible for the core tasks such as task sending and receiving, data standardization, and client management. Its main content is to coordinate and manage the information transmission between various modules and provide a stable and efficient data capture and storage service;

[0123] The client is used to initiate an HTTP request to crawl the corresponding page immediately after receiving the crawling task;

[0124] Specifically, after receiving a crawling task, it is used to construct a complete request message, including common basic information of the device and identification information specific to the site or app, and maintain the integrity and authenticity of the access interface service;

[0125] Downloader, responsible for sending requests and setting various response results, including maximum timeout processing, various encoding processing, exception callback, etc.

[0126] The storage device is used to extract, filter, clean and persist the captured data into the database according to the rules. It supports various relational and non-relational databases, and also supports the export of local text files (txt, excel, csv, etc.).

[0127] Exemplarily, the crawler implementation system for breaking through the server interface access restriction provided by the embodiment of the present invention includes:

[0128] The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed;

[0129] The sending module is used to send the captured data to the crawler scheduling server and mark the task ID. The captured data is the string returned by the HTTP response.

[0130] Preferably, the crawler implementation system that breaks through the server interface access restrictions runs on a server, and the server is used to provide a user input interface to implement the functions of the crawler implementation system that breaks through the server interface access restrictions when executed on an electronic device.

[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] Since the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention, their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0133] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment.

[0134] An embodiment of the present invention further provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0135] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0136] An embodiment of the present invention further provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, and switches.

[0137] An embodiment of the present invention further provides a server, which is used to provide a user input interface to implement the steps in the above method embodiments when executed on an electronic device.

[0138] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0140] To further illustrate the effects of the embodiments of the present invention, the following tests were conducted. The top 100 popular domestic websites were compared by frequently sending requests. The results showed that when the number of content sending requests exceeded 500 in one hour through direct access, 24% of the websites had data acquisition anomalies; when the number of requests exceeded 500 in one hour through this patented technology, only 3% of the websites had data acquisition anomalies, and the probability of being detected was reduced by more than 87%.

[0141] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A crawler implementation method that breaks through the server interface access restrictions, It is characterized in that This method generates page crawling tasks in batches and distributes them to the client. It relies on the device fingerprint and IP provided by the client to break through the access restrictions of the crawler scheduling server and maintain stable, sustainable and scalable data crawling. Specifically, it includes the following steps: S1, the crawler scheduling server takes tasks from the task pool, checks the online status of the client in sequence, and sends tasks to the client in sequence; S2, the client generates a complete request based on the task information and gives it to the downloader; S3, the downloader processes the request and sets the response processing results including maximum timeout processing, various encoding processing, and exception callback processing, and returns the results to the crawler scheduling server; S4, the crawler dispatches the server to extract and clean the data into the storage; S5, the memory is stored in a database or exported as a text according to needs.

2. The crawler implementation method for breaking through the server interface access restriction according to claim 1, It is characterized in that In step S1, the crawler scheduling server collects information about each task, including target link, name, periodicity and one-time settings, maximum time for each request, maximum number of visits, and whether to switch clients; the above information is subjected to the md5 digest algorithm to obtain a value, which is stored in the task pool. Based on the set characteristics, the uniqueness of each task is guaranteed; The crawler scheduling server starts to check the clients connected to the crawler scheduling server in turn, and issues tasks to the online clients in a round-robin manner to ensure that tasks are evenly distributed to each client.

3. The crawler implementation method for breaking through the server interface access restriction according to claim 1, It is characterized in that In step S2, after obtaining the task, the client constructs a complete request body based on the task information; the task information includes basic information such as UA and cookies, device fingerprint, site IP information, and app identification information; sends it to the downloader and marks the task as being in a waiting state.

4. The crawler implementation method for breaking through the server interface access restriction according to claim 1, It is characterized in that In step S3, after the downloader gets the request body, it sets the request timeout and the maximum number of attempts, sets an exception callback method for each task to avoid abnormal termination of system operation, sends the request and returns data to the crawler scheduling server through a standardized interface, marks the task execution status according to the response status code, and returns the task to another queue in the task pool for the task with the maximum number of failures, waiting for the next trigger execution; Among them, whether a task is fixed on the current client is determined by the crawler scheduling server. If no fixed client is set for execution when the task is created, the success rate of the client executing the task is determined.

5. The crawler implementation method for breaking through the server interface access restriction according to claim 4, It is characterized in that Set the exception callback method, including: Step 1: construct a two-dimensional callback model for different tasks, perform forward modeling to obtain normal operation data, and construct a task data set based on the obtained normal operation data; Step 2: construct an inversion network that runs normally, and preprocess and divide the constructed task data set; The normal inversion network has five convolution kernels, three pooling layers, and two fully connected layers; the convolution kernel size of the first convolution layer is 1×11, the convolution kernel size of the second convolution layer is 1×5, and the convolution kernel sizes of the other three convolution layers are all 1×3, and the step size is 1; the size of the three pooling layers is 1×3, and the step size is 2, all of which are performed in the VALID mode; the preprocessing and division of the constructed task data set includes: taking the first 80% of the data set as a training set and the last 20% as a test set; Step 3, using the processed task data set to train, optimize, and verify the constructed normal operation inversion network to obtain a trained normal operation inversion network; First, set the parameters and use the divided training set to train the AlexNet-Gra network; use the test set to verify whether the AlexNet-Gra network achieves the expected effect; if not, modify the parameters and train again; if the trained network achieves the expected effect, input the verification set data and reconstruct the two-dimensional callback model; the modified parameters include: (1) Select an optimization function to optimize the weights and biases that need to be updated in the network; the expression of the optimization function is: In the formula, are the weights and bias optimization values ​​that need to be updated in the network, m is the two-dimensional callback model, Θ is the weights and biases that need to be updated in the network, L() is the MSE loss function, Net() is the convolutional neural network that represents the mapping from the observed data d to the predicted two-dimensional callback model, and d is the observed data; (2) Select the objective function, the expression is: In the formula, L' is the objective function, N is the total number of data fitting, i is the number of data fitting, d i is the observed data of the i-th data fitting number, G is the M×N dimensional kernel matrix, m' i is the two-dimensional callback model of the fitting of the i-th data fitting number, is the weight regularization value, L(m,m') is the degree of model recovery, λ is the regularization parameter, n is the size of the training set, is the degree of data fitting, is weight decay regularization; Step 4: Input the normal operation data into the trained normal operation inversion network to obtain the inverted task abnormal termination result data.

6. The crawler implementation method for breaking through the server interface access restriction according to claim 5, It is characterized in that In step 1, a two-dimensional callback model for different tasks is constructed, including: The task space is divided into 800 rectangular cells with 20 rows × 40 columns. The size of each cell is 25ms × 25ms. The task nodes are set to multiple, and the transmission interval between task nodes is 10ms. Seven regular and single models are constructed, namely 75ms × 75ms, 100ms × 100ms, 125ms × 125ms, 75ms × 150ms, 150ms × 75ms, 100ms × 200ms, and 200ms × 100ms.

7. The crawler implementation method for breaking through the server interface access restriction according to claim 5, It is characterized in that In step 1, forward modeling is performed to obtain normal operation data, including: The matrix equation is used to represent the mapping relationship between the two-dimensional callback model m and the normal operation d, and different callback values ​​are set for the model task to perform forward modeling and generate corresponding data; The matrix equation is as follows: d=Gm d=(d 1 ,d 2 ,……,d M ) T m=(m 1 ,m 2 ,……,m N ) T Where T is the order, d M is the observed data of the number of task nodes, m N is a two-dimensional callback model with the number of grid units, G is an M×N dimensional kernel matrix, M is the number of task nodes, and N is the number of grid units.

8. The crawler implementation method for breaking through the server interface access restriction according to claim 1, It is characterized in that In step S4, after obtaining the data returned by the downloader, the crawler scheduling server extracts, filters, cleans the data, and sends it to the storage. If the task has not reached the maximum number of requests and the data returned by the downloader is abnormal, it is determined whether to switch the client to continue the task based on the task setting information.

9. The crawler implementation method for breaking through the server interface access restriction according to claim 1, It is characterized in that In step S5, the memory obtains the data sent by the crawler scheduling server, stores or updates the current data according to various types of relational and non-relational database APIs, and supports online export of local text files according to the task ID.

10. A crawler implementation system that breaks through the server interface access restrictions, It is characterized in that The system implements the crawler implementation method for breaking through the server interface access restriction as described in any one of claims 1 to 9, and the system includes: The task pool is used for users to store crawling target information, including URL, task ID, site name, task execution status information, timeout, and maximum number of attempts; The crawler scheduling server is used to issue a crawling task, which includes the task ID, the URL of the HTTP request, all parameters, and the maximum waiting time; The client is used to initiate an HTTP request to crawl the corresponding page immediately after receiving the crawling task; Downloader, used to send requests and set various response results, including maximum timeout processing, various encoding processing, and exception callback; Storage, which is used to extract, filter, clean and persist the captured data into the database according to rules. It supports various relational and non-relational databases, as well as local text file export. The detection module is used to check whether the maximum waiting time has been exceeded after the page crawling is completed; The sending module is used to send the captured data to the crawler scheduling server and mark the task ID. The captured data is the string returned by the HTTP response.

Images