A method and device for selecting a crawling time frequency of a web crawler

Abstract

The application relates to a crawling time frequency selection method and device of a web crawler, and belongs to the technical field of machine learning. The method comprises the following steps: judging whether a target website has an anti-crawling strategy, and if yes, obtaining a crawling time frequency critical value of the target website; constructing training data, training a model by using the training data, and obtaining a time frequency selection model; and calling the time frequency selection model, outputting an optimal crawling time frequency according to the crawling time frequency critical value. The method can automatically select the optimal crawling time frequency for the target website, data can be crawled according to the optimal crawling time frequency, the efficiency of data crawling of the web crawler is improved, and the work intensity of technical personnel is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine learning, and particularly relates to a web crawler crawling time frequency selection method and device. BACKGROUND

[0002] With the increasing development of Internet technology, more and more data are open on the Internet, and many of the data are exactly what is needed in daily work and study. If all the open data is manually sorted, it is time-consuming, labor-intensive and prone to errors. Therefore, a crawler technology is often used to automatically extract the data needed in the webpage.

[0003] However, for many companies, data is their core competence. If the data is freely and massively grabbed in the Internet environment, the companies will lose their core competitiveness. Therefore, most websites will formulate corresponding anti-crawler strategies. Crawler engineers often fail to perform data crawling tasks due to anti-crawler strategies. The crawling task of many complex websites needs experienced engineers to write, and the work intensity is large, and the website data crawling efficiency is low. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and provide a web crawler crawling time frequency selection method and device. The method can automatically select the optimal crawling time frequency for a target website, and perform data crawling according to the optimal crawling time frequency, thereby improving the efficiency of crawler data crawling and reducing the work intensity of technical personnel.

[0005] According to one aspect of the present application, the present application provides a web crawler crawling time frequency selection method, which comprises the following steps:

[0006] S1: judging whether the target website has an anti-crawling strategy, and if so, obtaining a crawling time frequency threshold value of the target website;

[0007] S2: constructing training data, training a model using the training data, and obtaining a time frequency selection model;

[0008] S3: calling the time frequency selection model, and outputting an optimal crawling time frequency according to the crawling time frequency threshold value.

[0009] Preferably, the judgment of whether the target website has an anti-crawling strategy comprises:

[0010] The data is crawled at a plurality of different time frequencies by requesting a URL, and the data is continuously crawled for a preset time; a response result is obtained, and whether the target website has an anti-crawling strategy is judged according to an HTTP status code in the response result or whether the response result contains normal data.

[0011] Preferably, the acquisition of the target website's crawling time frequency threshold value comprises:

[0012] Select an IP that has not been blocked from the IP address library of the target website, crawl data at multiple different time frequencies that gradually increase, and record the execution results of each request until the IP is blocked, to obtain the critical value of the crawling time frequency.

[0013] Preferably, the construction of training data, the training of the model using the training data, and the obtaining of the time frequency selection model comprise:

[0014] Obtain the service end IP list corresponding to the domain name of the target website through a domain name query service, take the service end IP address, the crawling time frequency, and the crawling state as training data, and train the model;

[0015] The model is a gradient boosting decision tree (GBDT) model, the service end IP address and the crawling time frequency in the training data are taken as features, and the crawling state is taken as a label, the GBDT model is trained, and the time frequency selection model is obtained.

[0016] Preferably, the calling of the time frequency selection model, the output of the optimal crawling time frequency according to the crawling time frequency threshold value comprises:

[0017] The time frequency selection model is called, the optimal crawling time frequency is output according to the crawling time frequency threshold value by using a binary search method.

[0018] According to another aspect of the present application, the present application also provides a crawling time frequency selection device of a web crawler, the device comprising:

[0019] A judgment module is configured to judge whether the target website has an anti-crawling strategy, and if so, to acquire the crawling time frequency threshold value of the target website.

[0020] A training module is configured to construct training data, train a model using the training data, and obtain a time frequency selection model.

[0021] An output module is configured to call the time frequency selection model, and output the optimal crawling time frequency according to the crawling time frequency threshold value.

[0022] Preferably, the judgment module judges whether the target website has an anti-crawling strategy.

[0023] The data is crawled at different time frequencies by requesting a URL, and the data is crawled for a preset time; a response result is obtained, and whether the target website has an anti-crawling strategy is determined according to an HTTP status code in the response result or whether normal data is contained in the response result.

[0024] Preferably, the judgment module obtains the crawling time frequency threshold value of the target website, including:

[0025] An IP that is not blocked is selected from an IP address library of the target website, data is crawled at a plurality of gradually increasing different time frequencies, and an execution result of each request is recorded until the IP is blocked, so as to obtain the crawling time frequency threshold value.

[0026] Preferably, the training module constructs training data, trains the model by using the training data, and obtains the time frequency selection model, including:

[0027] The domain name query service is used to obtain a service end IP list corresponding to the domain name of the target website, the service end IP address, the crawling time frequency, and the crawling state are used as training data, and the model is trained;

[0028] The model is a gradient boosting decision tree (GBDT) model, the service end IP address and the crawling time frequency in the training data are used as features, and the crawling state is used as a label, the GBDT model is trained, and the time frequency selection model is obtained.

[0029] Preferably, the output module calls the time frequency selection model, and outputs the optimal crawling time frequency according to the crawling time frequency threshold value, including:

[0030] The time frequency selection model is called, the optimal crawling time frequency is output by using a binary search method according to the crawling time frequency threshold value.

[0031] Beneficial effects: the present application can automatically select the optimal crawling time frequency for the target website by judging whether the target website has an anti-crawling strategy, obtaining the crawling time frequency threshold value of the target website in the case of yes, training the model by using the training data, obtaining the time frequency selection model, calling the time frequency selection model, and outputting the optimal crawling time frequency, crawling data according to the optimal crawling time frequency, improving the efficiency of the crawler in crawling data, and reducing the work intensity of the technical personnel.

[0032] The features and advantages of the present application will become clear through the following drawings and detailed description of the specific embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a crawling time frequency selection method flowchart of a web crawler.

[0034] Figure 2 is a schematic diagram of a crawling time frequency selection system of a web crawler. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be apparently and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0036] Embodiment 1

[0037] Figure 1 is a flowchart of a crawling time frequency selection method of a web crawler. As shown in the figure, the embodiment provides a crawling time frequency selection method of a web crawler, and the method comprises the following steps: Figure 1

[0038] S1: judging whether the target website has an anti-crawling strategy, and if yes, obtaining a crawling time frequency threshold value of the target website.

[0039] S2: constructing training data, training a model by using the training data, and obtaining a time frequency selection model;

[0040] S3: calling the time frequency selection model, outputting an optimal crawling time frequency according to the crawling time frequency threshold value.

[0041] Through the method of the embodiment, the optimal crawling time frequency for the target website can be automatically selected, and the data is crawled according to the optimal crawling time frequency, so that the efficiency of the crawler in crawling data is improved, and the work intensity of the technical personnel is reduced.

[0042] Preferably, the judging whether the target website has an anti-crawling strategy comprises:

[0043] The data is crawled at multiple different time frequencies by means of requesting a URL, and the data is continuously crawled for a preset time; a response result is obtained, and whether the target website has an anti-crawling strategy is judged according to an HTTP status code in the response result or whether the response result contains normal data.

[0044] Specifically, after the user determines the website to be crawled, the user inputs the website URL or the data interface URL.

[0045] ​Based on the rule, the target URL is accessed to determine whether the website URL has an anti-crawler strategy. The requests library of Python3 is used to request the URL, and data is crawled at a frequency of 1, 10, 100, and 1000 times per second, respectively, for 1 hour. If the data can be normally obtained, it is determined that the website basically has no anti-crawler strategy, and the time frequency of 1000 times / second is returned. If the response code is not 200 (HTTP status code exception), or the response content does not return normal data, it means that the website has an anti-crawler strategy.

[0046] Preferably, the acquisition of the crawling time frequency threshold of the target website comprises:

[0047] An IP that has not been blocked is selected from the IP address library of the target website, data is crawled at a plurality of different time frequencies that gradually increase, and the execution result of each request is recorded until the IP is blocked, to obtain the crawling time frequency threshold.

[0048] Specifically, if the HTTP status code is abnormal or there is no response after the request, it means that the website has an anti-crawler strategy. At this time, it is necessary to explore what frequency of data crawling will block the IP address of the target website. First, an IP that has not been blocked is selected from the IP library, and data is crawled at a frequency of 1, 2, 3 times per second, which gradually increases, for 1 hour, and the execution result of each request is recorded until the IP is blocked, which is the maximum value of the crawling frequency. The final result may be X times / second (the value of X may be 50, 100, 200, etc.), and the crawling frequency of the website is between 1 time / second and X times / second.

[0049] Preferably, the construction of the training data, the training of the model by using the training data, and the obtaining of the time frequency selection model comprise:

[0050] The service end IP list corresponding to the domain name of the target website is obtained through a domain name query service, the service end IP address, the crawling time frequency, and the crawling state are used as training data, and the model is trained;

[0051] The model is a gradient boosting decision tree (GBDT) model, the service end IP address and the crawling time frequency in the training data are used as features, and the crawling state is used as a label, the GBDT model is trained, and the time frequency selection model is obtained.

[0052] Specifically, since most websites are now deployed in clusters and have DNS acceleration strategies, the service end IP accessed each time may be different when accessing a URL. The service end IP list corresponding to the domain name can be obtained by using the Alibaba domain name query service.

[0053] The maximum value of the crawling frequency has been obtained through the above steps. Due to the inconsistent hardware conditions of the server and the existence of network fluctuations and other problems, if the maximum value is directly used to crawl data, the crawler will be unstable in most cases. At this time, a stable crawling frequency needs to be determined through a model algorithm.

[0054] This embodiment uses the binary search method to access the target IP several times, and records the server IP address, time frequency, crawling state (whether blocked) and other information of each time as the training data for binary classification.

[0055] Suppose the maximum value X of the frequency obtained in step 3 is 10, and the server IP list of a target website is 39.156.66.10, 39.156.66.11, 39.156.66.12. Then the training set data is:

[0056] Server IP address Crawl frequency (n / sec) Blocked or not 39.156.66.10 5 0, No 39.156.66.10 7 0, No 39.156.66.10 10 1, Yes 39.156.66.11 5 0, No 39.156.66.11 7 1, Yes 39.156.66.11 10 1, Yes 39.156.66.12 5 1, Yes 39.156.66.12 7 1, Yes 39.156.66.12 10 1, Yes

[0057] The server IP address and time frequency in the data set are used as features, and the crawling state (0 and 1 whether blocked) is used as a label. Based on GBDT (Gradient Boosting Tree), binary classification is performed. Python and pandas library are used to implement GBDT binary classification algorithm.

[0058] Machine learning can solve many problems, of which the two most important are regression and classification. GBDT can be used for regression problems. Compared with logistic regression, which can only be used for linear regression, GBDT can be used for linear and nonlinear regression, and its application is very wide. GBDT can also be used for binary classification problems (set threshold, greater than threshold is positive example, otherwise is negative example).

[0059] The specific algorithm implementation in this project is as follows:

[0060] 1) Parameter setting

[0061] Learning rate: learning_rate = 0.1

[0062] Number of iterations: n_trees = 5

[0063] Tree depth: max_depth = 3

[0064] 2) Initialize weak learner

[0065]

[0066] Where, arg min is the value of the variable when the following formula reaches the minimum value; x is the sample variable, which here refers to the characteristic value obtained by combining the IP address and the crawling frequency; c is the mean value of all training sample label values, which here refers to the mean value of the results of whether being blocked in all training data sets, N represents that the data set is divided into N units, y i is expressed as a predicted value.

[0067] The loss function L is a square loss, because the square loss function is a convex function, and the derivative is obtained by directly deriving:

[0068]

[0069] Let the derivative be equal to 0, and obtain:

[0070]

[0071] When initialized, c takes the mean value of all training sample label values (whether being blocked). At this time, the initial learner f0(x) is obtained.

[0072] 3) Multiple iterations

[0073] Since we set the number of iterations: n_trees = 5, M = 5 here. Here, M is the number of iterations or the number of iteration rounds, and M = 5 indicates that a total of 5 iterations are performed.

[0074] The negative gradient is calculated, and according to the square loss of the loss function in the foregoing, the negative gradient is the residual error.

[0075] At this time, the residual error is taken as the true value of the sample to train the weak learner.

[0076] Server IP address Crawl frequency (n / sec) Label value 39.156.66.10 5 0.875 39.156.66.10 7 -0.532 39.156.66.10 10 -0.243 39.156.66.11 5 0.744 39.156.66.11 7 -0.654 39.156.66.11 10 -0.262 39.156.66.12 5 0.912 39.156.66.12 7 -0.816 39.156.66.12 10 -0.533

[0077] 4) The final strong learner is:

[0078]

[0079] Where, γ jm is a fitting step, which refers to the predicted value of each tree; I(x∈R jm ) indicates an indicator function, and if x∈R jm) , I = 1, otherwise I = 0, R jm is a set of leaf node predicted values in all partition methods of the mth sub-tree, that is, the predicted values that can be reached by the mth regression tree. Wherein, the range of j is 1, 2, 3, …, J.

[0080] After the model training is completed, the model can also be verified by using test data in the embodiment, and the specific process includes:

[0081] A test data set is constructed, the service end IP address 39.156.66.12 is combined with a time frequency of 1 to 1000 times per second to form a test data set, and the test data is used to verify the model.

[0082] Server IP address Crawl frequency (n / sec) Model result 39.156.66.12 500 39.156.66.12 750 39.156.66.12 875 39.156.66.12 ...

[0083] Preferably, the calling the time frequency selection model according to the crawling time frequency threshold outputs the optimal crawling time frequency includes:

[0084] The calling the time frequency selection model according to the crawling time frequency threshold outputs the optimal crawling time frequency by using a binary search method.

[0085] Specifically, the model prediction result is obtained by using a binary search method. The middle value is taken to determine whether it is blocked, and if it is not blocked, the middle value is continuously taken until the blocking is obtained. Taking the maximum crawling time frequency threshold of 1000 times per second as an example, the process is as follows:

[0086] 1) Taking the middle value of 1-1000 times per second as an example, the model returns a result of not being blocked;

[0087] 2) Then continue to take the middle value of 500-1000 times per second, 750 times per second, and obtain the model result,

[0088] 3) If it is still not blocked, continue to take 875 times per second to obtain the model result;

[0089] 4) If the model returns a result of being blocked at this time, continue to take 750-875 as the minimum maximum value, and obtain the unblocked crawling frequency in the order from large to small according to the binary query;

[0090] 5) The obtained result is the final crawling frequency, and the frequency returned at this time is the final result of the device, and the final crawling time frequency of the blocking and unblocking threshold is returned.

[0091] The embodiment can automatically select the optimal crawling time frequency for the target website by judging whether the target website has an anti-crawling strategy, obtaining the crawling time frequency threshold of the target website in the case of yes, and training the model by using the training data to obtain the time frequency selection model, calling the time frequency selection model, and outputting the optimal crawling time frequency. According to the optimal crawling time frequency, the data is crawled, the efficiency of the crawler to crawl data is improved, and the work intensity of the technical personnel is reduced.

[0092] Embodiment 2

[0093] Figure 2 is a schematic diagram of a crawling time frequency selection system of a web crawler. AsFigure 2 The embodiment also provides a device for selecting a crawling time frequency of a web crawler, as shown in the figure. The device comprises:

[0094] a judging module 201 configured to judge whether a target website has an anti-crawling strategy, and obtain a crawling time frequency threshold of the target website if the target website has the anti-crawling strategy;

[0095] a training module 202 configured to build training data, train a model by using the training data, and obtain a time frequency selection model;

[0096] an output module 203 configured to call the time frequency selection model, and output an optimal crawling time frequency according to the crawling time frequency threshold.

[0097] Preferably, the judging module 201 judging whether a target website has an anti-crawling strategy comprises:

[0098] requesting a URL to crawl data at a plurality of different time frequencies, and continuing for a preset time; obtaining a response result, and judging whether the target website has an anti-crawling strategy according to an HTTP status code in the response result or whether the response result contains normal data.

[0099] Preferably, the judging module 201 obtaining a crawling time frequency threshold of the target website comprises:

[0100] selecting an IP that has not been blocked from an IP address library of the target website, crawling data at a plurality of different time frequencies that gradually increase, and recording an execution result of each request until the IP is blocked, to obtain a crawling time frequency threshold.

[0101] Preferably, the training module 202 building training data, training a model by using the training data, and obtaining a time frequency selection model comprises:

[0102] obtaining a service end IP list corresponding to a domain name of a target website by using a domain name query service, taking the service end IP address, the crawling time frequency, and the crawling state as training data, and training a model;

[0103] the model is a gradient boosting decision tree (GBDT) model, the service end IP address and the crawling time frequency in the training data are taken as features, the crawling state is taken as a label, the GBDT model is trained, and the time frequency selection model is obtained.

[0104] Preferably, the output module 203 calling the time frequency selection model, and outputting an optimal crawling time frequency according to the crawling time frequency threshold comprises:

[0105] Call the time frequency selection model, according to the crawling time frequency threshold, the binary search method is used to output the optimal crawling time frequency.

[0106] The specific process of the functions realized by each module in this embodiment 2 is the same as the implementation process of each step in embodiment 1, and will not be repeated here.

[0107] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made under the concept of the present application, using the content of the present application specification and drawings, or directly / indirectly applied in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for selecting the crawling time frequency of a web crawler, characterized in that, The method comprises the following steps: S1: judging whether the target website has an anti-crawling strategy, and if so, obtaining a crawling time frequency threshold value of the target website; S2: constructing training data, training a model using the training data, and obtaining a time frequency selection model; S3: calling the time frequency selection model, and outputting an optimal crawling time frequency according to the crawling time frequency threshold value; The method comprises the following steps: Selecting an IP that has not been blocked from an IP address library of the target website, crawling data at a plurality of different time frequencies that gradually increase, and recording the execution result of each request until the IP is blocked, to obtain a crawling time frequency threshold value; The method comprises the following steps: Obtaining a service end IP list corresponding to the domain name of the target website through a domain name query service, taking the service end IP address, the crawling time frequency, and the crawling state as training data, and training the model; The model is a gradient boosting decision tree (GBDT) model, the service end IP address and the crawling time frequency in the training data are taken as features, and the crawling state is taken as a label to train the GBDT model, to obtain the time frequency selection model.

2. The method of claim 1, wherein, The method comprises the following steps: Crawling data at a plurality of different time frequencies by requesting a URL, and continuing for a preset time; obtaining a response result, and judging whether the target website has an anti-crawling strategy according to the HTTP status code in the response result or whether the response result contains normal data.

3. The method of claim 1, wherein, The method comprises the following steps: Calling the time frequency selection model, and outputting an optimal crawling time frequency according to the crawling time frequency threshold value by using a binary search method.

4. A crawling time frequency selection apparatus of a web crawler, characterized in that, The device comprises: A judgment module for judging whether the target website has an anti-crawling strategy, and if so, obtaining a crawling time frequency threshold value of the target website; A training module for constructing training data, training a model using the training data, and obtaining a time frequency selection model; An output module for calling the time frequency selection model, and outputting an optimal crawling time frequency according to the crawling time frequency threshold value; The judgment module obtains the crawling time frequency threshold value of the target website by the following steps: Selecting an IP that has not been blocked from an IP address library of the target website, crawling data at a plurality of different time frequencies that gradually increase, and recording the execution result of each request until the IP is blocked, to obtain a crawling time frequency threshold value; The training module constructs training data, trains a model using the training data, and obtains a time frequency selection model by the following steps: Obtaining a service end IP list corresponding to the domain name of the target website through a domain name query service, taking the service end IP address, the crawling time frequency, and the crawling state as training data, and training the model; The model is a gradient boosting decision tree (GBDT) model, the server IP address and the crawling time frequency in the training data are taken as features, the crawling state is taken as a label, the GBDT model is trained, and the time frequency selection model is obtained.

5. The apparatus of claim 4, wherein, The judgment module judges whether the target website has an anti-crawling strategy, which includes: Data is crawled at multiple different time frequencies by requesting a URL, and the data is continuously crawled for a preset time; a response result is obtained, and whether the target website has an anti-crawling strategy is judged according to an HTTP status code in the response result or whether the response result contains normal data.

6. The apparatus of claim 4, wherein, The output module calls the time frequency selection model, and outputs an optimal crawling time frequency according to the crawling time frequency threshold, which includes: calling the time frequency selection model, and outputting an optimal crawling time frequency by using a binary search method according to the crawling time frequency threshold.

Images