A method and device for detecting crawler requests
By building session sequences and information groups of different dimensions, generating feature vectors, and training crawler detection models, the problem that traditional detection rules cannot recognize new crawler requests is solved, and accurate detection of traditional and new crawler requests is achieved, reducing the load of website servers pressure.
Patent Information
- Application Number
- CN201911281120.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-12-13
AI Technical Summary
Traditional crawler request detection rules cannot effectively identify requests from new web crawlers, resulting in the load pressure of website servers still exists, and a detection solution that can identify traditional and new crawler requests is urgently needed.
By obtaining the historical access information of the target website, it is divided into session sequences and information groups of different dimensions, generating feature vectors, training crawler detection models, and using machine learning technology to detect crawler requests.
Accurate detection of traditional and new crawler requests is achieved, reducing the load pressure of the website server, and improving detection efficiency and accuracy.
Smart Images

Figure CN112989157B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network communication technology, and in particular to a method and device for detecting crawler requests. Background Art
[0002] A web crawler is a program or script that automatically grabs data resources from a website according to preset rules. It can start from one or several web pages to grab resources and links in the web pages by continuously sending access requests (called crawler requests) to the website server, and then continue to access and crawl subsequent web pages through the crawled links until all the required web pages are crawled.
[0003] According to statistics, crawler requests currently account for more than half of the access requests to web pages. For website operators, a large number of crawler requests will cause extremely high load pressure on the website server. Therefore, website operators will adopt certain crawler detection schemes to filter out crawler requests from all access requests to web pages. Specifically, the received access requests can be detected based on pre-set crawler request detection rules. For example, when the source IP address of an access request belongs to an IP address in the blacklist, or a client accesses all pages of a web link in a session, it can be determined that the corresponding access request is a crawler request, or the corresponding client is the sender of the crawler request.
[0004] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:
[0005] With the development of computer technology, new web crawlers have emerged that meet the characteristics of cross-session, random crawling frequency, and application IP proxy pool. Traditional crawler request detection rules can no longer effectively detect the new crawler requests generated by these new web crawlers. Therefore, there is an urgent need for a crawler detection solution that can not only identify traditional crawler requests but also effectively detect new crawler requests. Summary of the invention
[0006] In order to solve the problems of the prior art, an embodiment of the present invention provides a method and device for detecting crawler requests. The technical solution is as follows:
[0007] In a first aspect, a method for detecting a crawler request is provided, the method comprising:
[0008] Obtain historical access information of the target website in the target historical period, and classify the historical access information containing the same main dimension identifier into the same information group;
[0009] Based on the historical access information under the target information group, create a plurality of single-dimensional conversation sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional conversation sequence corresponding to the primary dimension identifier;
[0010] Generate a feature vector corresponding to the target information group according to the traffic characteristics corresponding to the cross-dimensional conversation sequence and the traffic characteristics corresponding to the multiple single-dimensional conversation sequences;
[0011] Based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model.
[0012] In a second aspect, a device for detecting a crawler request is provided, the device comprising:
[0013] An information acquisition module, used to acquire historical access information of a target website in a target historical period, and to classify the historical access information containing the same main dimension identifier into the same information group;
[0014] A sequence creation module, used to create a plurality of single-dimensional conversation sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional conversation sequence corresponding to the primary dimension identifier based on the historical access information under the target information group;
[0015] A feature extraction module, used to generate a feature vector corresponding to the target information group according to the traffic features corresponding to the cross-dimensional conversation sequence and the traffic features corresponding to the multiple single-dimensional conversation sequences;
[0016] The crawler detection module is used to train a crawler detection model corresponding to the target website based on the feature vectors corresponding to all information groups under the target website, and perform crawler detection on the target website through the crawler detection model.
[0017] In a third aspect, a network device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for detecting crawler requests as described in the first aspect.
[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for detecting crawler requests as described in the first aspect.
[0019] The technical solution provided by the embodiment of the present invention has the following beneficial effects:
[0020] In an embodiment of the present invention, historical access information of a target website in a target historical period is obtained, and historical access information containing the same main dimension identifier is divided into the same information group; based on the historical access information under the target information group, multiple single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group and cross-dimensional session sequences corresponding to the main dimension identifier are created; according to the traffic characteristics corresponding to the cross-dimensional session sequence and the traffic characteristics corresponding to the multiple single-dimensional session sequences, a feature vector corresponding to the target information group is generated; based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model. In this way, by constructing session sequences of different dimensions, the access requests are analyzed as a whole in a cross-session manner, and then using machine learning technology to construct crawler detection models for different websites, the overall characteristics and sending rules of crawler requests can be discovered more intuitively and conveniently, so that traditional crawler requests and new crawler requests can be detected more accurately and effectively. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 is a flow chart of a method for detecting crawler requests provided by an embodiment of the present invention;
[0023] Figure 2 is a flow chart of a method for detecting crawler requests provided by an embodiment of the present invention;
[0024] Figure 3 is a flow chart of a method for detecting crawler requests provided by an embodiment of the present invention;
[0025] Figure 4 It is a schematic diagram of the structure of a device for detecting crawler requests provided by an embodiment of the present invention;
[0026] Figure 5 It is a structural diagram of a network device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0028] The embodiment of the present invention provides a method for detecting crawler requests, the execution subject of the method can be any network device with data processing function, and can create and train intelligent models based on machine learning technology. Among them, the network device can be the background server of any website, which can create an intelligent model for crawler detection based on the access history of the website, and then use the intelligent model to detect subsequent crawler requests. The network device can also be a network node for receiving and forwarding website access requests (such as a node server in a CDN cluster), which can train a crawler detection model for each website based on the historical reception of access requests, and then use the crawler detection model to detect crawler requests. The above-mentioned network device may include a processor, a memory and a transceiver, the processor can be used to perform the processing of detecting crawler requests in the following process, the memory can be used to store the data required and the data generated in the following processing process, and the transceiver can be used to receive and send the relevant data in the following processing process.
[0029] The following will be combined with specific implementation methods. Figure 1 The processing flow shown is described in detail, and the content can be as follows:
[0030] Step 101, obtaining historical access information of a target website in a target historical period, and classifying historical access information containing the same main dimension identifier into the same information group.
[0031] Among them, the main dimension identifier can be carried in the user's access request to the website, and can be used to identify different user terminals and distinguish different access requests. Specifically, it can be any one of the source IP address, user ID or device fingerprint in the access request, which can be selected according to actual needs. Of course, this embodiment also supports the selection of other feasible identifiers as the main dimension identifier.
[0032] In implementation, after the target website is online, the user can send an access request for the target website to the network device to access any web page in the target website. After receiving the access request for the target website, the network device can record the seven-tuple information of the access request, the URL pointed to, the access time, the data packet size and other parameter information to generate the historical access information of the target website. Afterwards, the network device can use the historical access information to train and generate the crawler detection model corresponding to the target website. Specifically, the network device can obtain the historical access information of the target website in the target historical period, such as obtaining the historical access information of the target website within the previous 7 days, and then group all the historical access information according to the pre-selected main dimension identifier to divide the historical access information containing the same main dimension identifier into the same information group. For example, the main dimension identifier is the source IP address, and the network device can group the historical access information of the target website within the previous 7 days according to the source IP address, and each information group is composed of historical access information containing the same source IP address, and different information groups can correspond to different source IP addresses. It is worth mentioning that in the early stage of the target website going online, since there is not enough historical access information to establish a crawler detection model, the network equipment can first use traditional crawler detection technology to perform crawler detection on the target website until a crawler detection model is established based on the historical access information.
[0033] Step 102: Based on the historical access information under the target information group, create multiple single-dimensional conversation sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional conversation sequence corresponding to the primary dimension identifier.
[0034] Among them, the secondary dimension identifier can be carried in the user's access request to the website. In addition to the primary dimension identifier, it can be further used to identify different user terminals and distinguish different access requests. It can be one or more of the source IP address, user ID, device fingerprint or browser identifier that are different from the primary dimension identifier. The specific identifier can be selected according to actual needs. Of course, this embodiment also supports the selection of other feasible identifiers as secondary dimension identifiers. The target information group is any information group obtained by dividing the historical access information in step 101.
[0035] In implementation, after the network device groups the historical access information according to the main dimension identifier, a session sequence can be created in each information group. The session sequence can be an ordered set consisting of multiple groups of session information, and each group of session information can contain at least one specific access information in an HTTP session. Taking the target information group as an example, the network device can create a cross-dimensional session sequence corresponding to the main dimension identifier and multiple single-dimensional session sequences corresponding to the secondary dimension identifier based on the historical access information under the target information group. Specifically, the network device can divide the historical access information under the target information group into multiple groups of session information based on the HTTP session granularity. After that, on the one hand, multiple groups of conversation information can be arranged in order to generate a cross-dimensional conversation sequence corresponding to the main dimension identifier; on the other hand, multiple groups of conversation information can be divided according to the secondary dimension identifier A, and the conversation information with the secondary dimension identifier A can be arranged in order to generate multiple single-dimensional conversation sequences corresponding to the secondary dimension identifier A; further, the secondary dimension identifier B can be selected to re-divide the multiple groups of conversation information, and then the conversation information with the secondary dimension identifier B can be arranged in order to generate multiple single-dimensional conversation sequences corresponding to the secondary dimension identifier B, so that multiple single-dimensional conversation sequences corresponding to all secondary dimension identifiers can be generated in sequence. Here, two secondary dimension identifiers can also be selected at the same time to divide and arrange the conversation information, and generate multiple single-dimensional conversation sequences corresponding to the two secondary dimension identifiers.
[0036] For example, the primary dimension is identified by the source IP address, the secondary dimension is identified by the user ID, browser identifier, and device fingerprint, and there are 30 groups of session information under the information group IP1. The 30 groups of session information can be arranged in order to generate a cross-dimensional session sequence under the information group IP1. If the 30 groups of session information include three user IDs "ID1, ID2, ID3", the 30 groups of session information can be divided and arranged by user ID to generate a single-dimensional session sequence corresponding to the three user IDs; then, if the 30 groups of session information include two device fingerprints "MAC1, MAC2", the 30 groups of session information can be divided and arranged by device fingerprint to generate a single-dimensional session sequence corresponding to the two device fingerprints; next, if the 30 groups of session information include four browser identifiers "UA1, UA2, UA3, UA4", the 30 groups of session information can be divided and arranged by browser identifier to generate a single-dimensional session sequence corresponding to the four browser identifiers.
[0037] Specifically, the creation process of the above-mentioned single-dimensional conversation sequence can be as follows: for the target sub-dimensional identifier under the target information group, obtain the web page access information contained in the conversation corresponding to each target sub-dimensional identifier information; arrange the web page access information according to the access time, and generate a single-dimensional conversation sequence corresponding to each target sub-dimensional identifier information.
[0038] Among them, the target sub-dimension identifier can be any pre-selected sub-dimension identifier, and the target sub-dimension identifier information can be the specific identification information under the target sub-dimension identifier. For example, the target sub-dimension identifier is the user ID, and the target sub-dimension identifier information is "ID1:893***221", "ID2:668***203", etc.
[0039] In implementation, taking the target sub-dimension identifier under the target information group as an example, when the network device creates multiple single-dimensional session sequences corresponding to the target sub-dimension identifier, it can first extract all the target sub-dimension identifier information from the historical access information under the target information group, and then filter the sessions corresponding to each target sub-dimension identifier information, and then obtain all the web page access information contained in these sessions. Among them, the web page access information may include at least the page identifier and access time of the web page. Afterwards, the network device can arrange the above web page access information in the order of access time, thereby generating a single-dimensional session sequence corresponding to each target sub-dimension identifier information. S can be defined as {s1, s2, s3,…, s n} is the session set under the target information group, element s i Represents a session. represents the jth visit in the i-th session, h is the page ID, and t is the access time. Assume that the sessions corresponding to the target secondary dimension ID information are: in, , then the one-dimensional conversation sequence can be
[0040] Specifically, the process of creating the above cross-dimensional conversation sequence may be as follows: all web page access information contained in all conversations of the target information group is arranged according to the access time, and a cross-dimensional conversation sequence corresponding to the main dimension identifier is generated.
[0041] The web page access information may include at least a page ID and access time of the web page.
[0042] In implementation, taking the target information group as an example, when the network device creates a cross-dimensional session sequence corresponding to the primary dimension identifier, it can arrange all web page access information contained in all sessions of the target information group in the order of access time, thereby generating a cross-dimensional session sequence corresponding to the primary dimension identifier. For example, based on the above processing of the single-dimensional session sequence, the target information group has a total of three single-dimensional session sequences of "SP1, SP2, SP3" for the target secondary dimension identifier, then each SP i Next By sorting by access time, the cross-dimensional conversation sequence corresponding to the main dimension identifier under the target information group can be obtained.
[0043] Step 103: Generate a feature vector corresponding to the target information group according to the traffic characteristics corresponding to the cross-dimensional conversation sequence and the traffic characteristics corresponding to multiple single-dimensional conversation sequences.
[0044] In implementation, after the network device creates multiple single-dimensional conversation sequences and cross-dimensional conversation sequences for the target information group, the traffic characteristics of each conversation sequence can be calculated. The traffic characteristics here can be pre-set and used to characterize the characteristics of multiple groups of conversation information contained in each conversation sequence, such as conversation interval time, request packet size, request type, request packet content and other characteristics. It is worth mentioning that for cross-dimensional conversation sequences, the similarity between the conversation information corresponding to different secondary dimension identifiers can also be detected. Next, the network device can integrate and splice the calculated traffic characteristics of each conversation sequence to generate a feature vector corresponding to the target information group.
[0045] Optionally, the conversation sequence can be scored according to the different website attributes carried in the conversation, and the scoring result can be used as a feature vector. The corresponding processing can be as follows: obtain the website map of the target website, and establish an attribute score library of the target website based on the website map; score each single-dimensional conversation sequence and cross-dimensional conversation sequence according to the attribute score library, and set the scoring result as the feature vector of the target information group.
[0046] The attribute score library records the score values of different website attributes of the target website. The website attributes at least include the webpage URL, the webpage referer, and all supported browser identifiers (ie, UA).
[0047] In implementation, the network device can obtain the website map of the target website, and then record the access and jump relationship of each web page in the target website and all browser identifiers supported by the target website according to the website map, and build an attribute score library. The website map here can be actively provided by the target website, or it can be created by the network device based on the historical access information of the target website; since the content of the website will be continuously updated, the network device can also regularly update the website map and the attribute score library. Specifically, different scoring mechanisms can be used to score the same website attribute, and then the scores obtained by different scoring mechanisms are weighted to obtain a comprehensive score for the website attribute. For example, there are three sets of scoring mechanisms. Scoring mechanism A is implemented based on the frequency of occurrence of network attributes on the website, scoring mechanism B is implemented based on the frequency of occurrence of network attributes in all dynamic pages, and scoring mechanism C is a manual evaluation of the attribute value based on business experience. For example, scoring mechanism A can have scores for the three website attributes of "URL, UA, referer": a url ={(url1, a_score url1), (url2, a_score url2 ), (url3, a_score url3 )}, a ua ={(ua1, a_score ua1 ), (ua2, a_score ua2 )}, a referer ={(referer1, a_score referer1 ), (referer2, a_score referer2 )}; Scoring mechanism B can have scores for two website attributes: "URL, UA": b url ={(url1, b_score url1 ), (url2, b_score url2 )}, b ua ={(ua1, b_score url1 ), (ua3, b_score ua3 )}; The same is true for scoring mechanism C or other scoring mechanisms. The weighted score for each website attribute can be: score_traget i =(a_score target_i + b_score target_i +…+n_score target_i ) / n. Furthermore, the network device can score each session according to the attribute score library, and then score each single-dimensional session sequence and cross-dimensional session sequence based on the session score, and can calculate the maximum score, minimum score, average score, weighted score, etc. in each session sequence, so that the scoring result can be set as the feature vector of the target information group. In this way, the relevance of the session and the website is evaluated from multiple angles through the scoring mechanism, and the session score is used as the detection angle of the crawler request, which can summarize the law of the crawler request with a higher probability and effectively improve the accuracy of crawler detection.
[0048] Optionally, the human-computer interaction information when the user visits the target website can also be used as a feature vector, and the corresponding processing can be as follows: based on the human-computer interaction information in the single-dimensional conversation sequence and the cross-dimensional conversation sequence, generate a feature vector corresponding to the target information group.
[0049] In implementation, after receiving the access request of the target website sent by the user end, the network device can embed a human-computer interaction detection program in the feedback message, which can be used to monitor whether the user end has mouse movement, clicks, keystrokes and other behaviors, and can report the detection results to the network device. In this way, the network device can record the human-computer interaction information generated in each session, and after creating a single-dimensional session sequence and a cross-dimensional session sequence, it can generate a feature vector corresponding to the target information group based on the human-computer interaction information contained in the above session sequence. In addition, the network device can combine the device fingerprint to further determine whether the detected human-computer interaction information contains forged information. In this way, by detecting the human-computer interaction information and determining the crawler request from the perspective of human-computer interaction, normal requests and crawler requests can be more effectively identified, and the accuracy of crawler detection can be improved.
[0050] Step 104: Based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model.
[0051] In implementation, the network device can generate feature vectors corresponding to all information groups under the target website according to the processing of step 102 and step 103. In this way, the network device can use machine learning technology to train the crawler detection model corresponding to the target website based on these feature vectors. Specifically, the network device can use a supervised algorithm or an unsupervised algorithm to train the crawler detection model. After the crawler detection model training is completed, the network device can perform crawler detection on the target website through the crawler detection model, that is, identify the crawler request in all access requests received from the target website. Further, since the content of the website often changes and the network crawler is constantly updated, it is necessary to set a certain validity period for the crawler detection model. When the network device uses the crawler detection model to perform crawler detection, it can first determine whether the current moment is within the validity period of the crawler detection model. If it is, the detection can continue, otherwise the processing of steps 101 to 104 can be re-executed to update the crawler detection model.
[0052] It is worth mentioning that when a new website is launched, or the crawler detection model of the website needs to be updated, or when it is found that the accuracy of the detection results of the crawler detection model is too low, the above steps 101 to 104 can be triggered to execute.
[0053] Optionally, there are various mechanisms for crawler detection using crawler detection models, such as the following: Figure 2 and Figure 3 Two possible mechanisms are shown:
[0054] Mechanism 1: Step 201, periodically counting the main dimension identification information corresponding to the target website appearing in the current period.
[0055] Among them, the main dimension identification information can be specific identification information under the main dimension identification. For example, the main dimension identification is the source IP address, and the main dimension identification information can be "IP address 1: 192.***.***.200", "IP address 2: 255.***.***.101", etc.
[0056] In implementation, the network device may periodically analyze the historical access information of the target website in the current period and count all the main dimension identification information appearing therein. For example, the network device may perform statistics every 10 minutes and obtain all the source IP addresses appearing in the historical access information of the target website in the previous 10 minutes.
[0057] Step 202: for each main dimension identification information, based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information.
[0058] The processing of this step may specifically refer to step 102, wherein the preset duration may be pre-set by a technician and may be appropriately adjusted according to demand, such as 6 hours, 12 hours, etc.
[0059] Step 203: Generate a feature vector corresponding to the main dimension identification information according to the traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information.
[0060] For details of the processing of this step, please refer to step 103.
[0061] Step 204: input the feature vector corresponding to the main dimension identification information into the crawler detection model, and determine whether the main dimension identification information belongs to a crawler request based on the model output content.
[0062] In this way, periodic batch detection of access requests can ensure the timeliness of feedback for normal access requests, and there is no need to frequently perform crawler detection processing, which can reduce the resource consumption of the device to a certain extent.
[0063] Mechanism 2: Step 301, when an access request from a target website is received, the main dimension identification information of the access request is obtained.
[0064] In practice, after receiving an access request from a target website, the network device can detect the access request, that is, it can first obtain the main dimension identification information of the access request. In this way, if the access request is detected to be a normal request, the access request can be responded to according to the conventional response mechanism; if the access request is detected to be a crawler request, the crawler request can be marked and discarded.
[0065] Step 302: Based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information.
[0066] The processing of this step may specifically refer to step 102, wherein the preset duration may be pre-set by a technician and may be appropriately adjusted according to demand, such as 6 hours, 12 hours, etc.
[0067] Step 303: Generate a feature vector corresponding to the main dimension identification information according to the traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information.
[0068] For details of the processing of this step, please refer to step 103.
[0069] Step 304: Input the feature vector corresponding to the main dimension identification information into the crawler detection model, and determine whether the access request is a crawler request based on the model output content.
[0070] In this way, each access request is detected, and crawler requests can be identified promptly and effectively, thus preventing the website content from being frequently maliciously crawled.
[0071] Optionally, after receiving an access request, it can be simply detected by comparing the dimension identification information. The corresponding processing can be as follows: if the similarity of all dimension identification information between the received target access request and the detected crawler request is greater than a preset threshold, the target access request is marked as a crawler request.
[0072] In implementation, each time the network device detects a crawler request, it can record all dimensional identification information of the crawler request. Therefore, after receiving the target access request, the network device can first obtain all dimensional identification information of the target access request, and then compare it with the dimensional identification information of the detected crawler request as a whole. If it is found that the overall similarity of all dimensional identification information between the target access request and a crawler request is greater than a preset threshold, the target access request can be directly identified as a crawler request without the need to use a crawler detection model to detect the target access request. In this way, by first performing a preliminary detection of the access request by comparing all dimensional identification information, some crawler requests can be quickly and easily identified, thereby reducing the workload of crawler detection and saving equipment resources.
[0073] Optionally, after the model training is completed, the detection accuracy of the crawler detection model can be verified in advance, and the corresponding processing can be as follows: verify the detection accuracy of the crawler detection model based on the preset crawler feature material library; if the detection accuracy is lower than the preset threshold, then supplement the new historical visit information of the target website; based on the new historical visit information and historical visit information, retrain the crawler detection model corresponding to the target website.
[0074] In implementation, a crawler feature material library may be pre-set at the network device, and the crawler feature material library may contain a large number of feature vectors of determined crawler requests. After the network device has trained the crawler detection model corresponding to the target website, it may verify the detection accuracy of the crawler detection model corresponding to the target website based on the crawler feature material library. If the detection accuracy is lower than the preset threshold, the network device may supplement the acquisition of the newly added historical access information of the target website, and the newly added historical access information may be the historical access information of the target website in addition to the historical access information acquired during the model training in step 101, that is, if the historical access information of the past 7 days is acquired during the training of the crawler detection model, the newly added historical access information may be the historical access information of the previous 8 days. Afterwards, the network device may retrain the crawler detection model corresponding to the target website based on the newly added historical access information and the historical access information, in the process of steps 101 to 104, until the detection accuracy of the crawler detection model reaches the preset threshold. In this way, on the one hand, verifying the crawler detection model through the crawler feature material library can effectively ensure the detection accuracy of the crawler detection model; on the other hand, supplementing the historical access information and retraining the crawler detection model can make the model closer to the crawler detection needs of the target website.
[0075] It should be noted that, given that crawler requests are beneficial to the promotion of website content, network devices can change the way the model is trained or set up a crawler request whitelist to avoid blocking the crawler detection model from detecting some crawler requests.
[0076] In an embodiment of the present invention, historical access information of a target website in a target historical period is obtained, and historical access information containing the same main dimension identifier is divided into the same information group; based on the historical access information under the target information group, multiple single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group and cross-dimensional session sequences corresponding to the main dimension identifier are created; according to the traffic characteristics corresponding to the cross-dimensional session sequence and the traffic characteristics corresponding to the multiple single-dimensional session sequences, a feature vector corresponding to the target information group is generated; based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model. In this way, by constructing session sequences of different dimensions, the access requests are analyzed as a whole in a cross-session manner, and then using machine learning technology to construct crawler detection models for different websites, the overall characteristics and sending rules of crawler requests can be discovered more intuitively and conveniently, so that traditional crawler requests and new crawler requests can be detected more accurately and effectively.
[0077] Based on the same technical concept, the embodiment of the present invention also provides a device for detecting crawler requests, such as Figure 4 As shown, the device comprises:
[0078] The information acquisition module 401 is used to acquire the historical access information of the target website in the target historical period, and classify the historical access information containing the same main dimension identifier into the same information group;
[0079] A sequence creation module 402 is used to create a plurality of single-dimensional conversation sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional conversation sequence corresponding to the primary dimension identifier based on the historical access information under the target information group;
[0080] A feature extraction module 403, configured to generate a feature vector corresponding to the target information group according to the traffic feature corresponding to the cross-dimensional conversation sequence and the traffic features corresponding to the multiple single-dimensional conversation sequences;
[0081] The crawler detection module 404 is used to train a crawler detection model corresponding to the target website based on the feature vectors corresponding to all information groups under the target website, and perform crawler detection on the target website through the crawler detection model.
[0082] Optionally, the crawler detection module 404 is specifically used to:
[0083] Periodically counting the main dimension identification information corresponding to the target website appearing in the current period;
[0084] For each main dimension identification information, based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information;
[0085] Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information;
[0086] The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the main dimension identification information belongs to a crawler request based on the model output content.
[0087] Optionally, the crawler detection module 404 is specifically used to:
[0088] When receiving an access request from a target website, obtaining primary dimension identification information of the access request;
[0089] Based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information;
[0090] Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information;
[0091] The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the access request is a crawler request based on the model output content.
[0092] Figure 5 5 is a schematic diagram of the structure of the network device provided by an embodiment of the present invention. The network device 500 may have relatively large differences due to different configurations or performances, and may include one or more central processors 522 (for example, one or more processors) and a memory 532, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the network device 500. Furthermore, the central processor 522 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the network device 500.
[0093] The network device 500 may also include one or more power supplies 529, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, one or more keyboards 556, and / or one or more operating systems 541, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.
[0094] The network device 500 may include a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the above-mentioned crawler request detection.
[0095] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for detecting crawler requests, characterized in that: The method comprises: Obtain historical access information of the target website in the target historical period, and classify the historical access information containing the same main dimension identifier into the same information group; Based on the historical access information under the target information group, multiple single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional session sequence corresponding to the primary dimension identifier are created; wherein the primary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is used to identify different user terminals and distinguish different access requests; the secondary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is in addition to the primary dimension identifier, and is further used to identify different user terminals and distinguish different access requests; Generate a feature vector corresponding to the target information group according to the traffic characteristics corresponding to the cross-dimensional conversation sequence and the traffic characteristics corresponding to the multiple single-dimensional conversation sequences; Based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model.
2. The method according to claim 1, characterized in that The primary dimension identifier is the source IP address, user ID or device fingerprint; the secondary dimension identifier is the source IP address, user ID, device fingerprint or browser identifier.
3. The method according to claim 1, characterized in that The step of creating a plurality of single-dimensional conversation sequences corresponding to the secondary-dimensional identifiers under the target information group includes: For the target sub-dimension identifier under the target information group, obtain the web page access information contained in the session corresponding to each target sub-dimension identifier information; The web page access information is arranged according to the access time to generate a single-dimensional session sequence corresponding to each target secondary-dimensional identification information.
4. The method according to claim 1, characterized in that: The step of creating a cross-dimensional conversation sequence corresponding to the main dimension identifier includes: All web page access information contained in all sessions of the target information group is arranged according to access time to generate a cross-dimensional session sequence corresponding to the main dimension identifier.
5. The method according to claim 1, characterized in that: The method further comprises: Obtaining a website map of the target website, and establishing an attribute score library of the target website based on the website map, wherein the attribute score library records the score values of different website attributes of the target website, and the website attributes at least include a webpage URL, a webpage referer, and all supported browser identifiers; Each of the single-dimensional conversation sequence and the cross-dimensional conversation sequence is scored according to the attribute score library, and the scoring result is set as a feature vector of the target information group.
6. The method according to claim 1, characterized in that The method further comprises: Based on the human-computer interaction information in the single-dimensional conversation sequence and the cross-dimensional conversation sequence, a feature vector corresponding to the target information group is generated.
7. The method according to claim 1, characterized in that The performing crawler detection on the target website by using the crawler detection model includes: Periodically counting the main dimension identification information corresponding to the target website appearing in the current period; For each main dimension identification information, based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the main dimension identification information belongs to a crawler request based on the model output content.
8. The method according to claim 1, characterized in that The performing crawler detection on the target website by using the crawler detection model includes: When receiving an access request from a target website, obtaining primary dimension identification information of the access request; Based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the access request is a crawler request based on the model output content.
9. The method according to claim 7 or 8, characterized in that: The method further comprises: If the similarity of all dimension identification information between the received target access request and the detected crawler request is greater than a preset threshold, the target access request is marked as a crawler request.
10. The method according to claim 1, characterized in that After establishing the crawler detection model corresponding to the target website, the following steps are also included: Verifying the detection accuracy of the crawler detection model based on a preset crawler feature material library; If the detection accuracy is lower than a preset threshold, additional historical visit information of the target website is obtained; Based on the newly added historical access information and the historical access information, the crawler detection model corresponding to the target website is retrained.
11. A device for detecting crawler requests, characterized in that: The device comprises: An information acquisition module, used to acquire historical access information of a target website in a target historical period, and to classify the historical access information containing the same main dimension identifier into the same information group; A sequence creation module, for creating, based on historical access information under a target information group, a plurality of single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group, and a cross-dimensional session sequence corresponding to the primary dimension identifier; wherein the primary dimension identifier is an identifier carried in an access request of a user terminal to the target website, and is used to identify different user terminals and distinguish different access requests; and the secondary dimension identifier is an identifier carried in an access request of the user terminal to the target website, and is in addition to the primary dimension identifier, and is further used to identify different user terminals and distinguish different access requests; A feature extraction module, used to generate a feature vector corresponding to the target information group according to the traffic features corresponding to the cross-dimensional conversation sequence and the traffic features corresponding to the multiple single-dimensional conversation sequences; The crawler detection module is used to train a crawler detection model corresponding to the target website based on the feature vectors corresponding to all information groups under the target website, and perform crawler detection on the target website through the crawler detection model.
12. The device according to claim 11, characterized in that The crawler detection module is specifically used for: Periodically counting the main dimension identification information corresponding to the target website appearing in the current period; For each main dimension identification information, based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the main dimension identification information belongs to a crawler request based on the model output content.
13. The device according to claim 11, characterized in that The crawler detection module is specifically used for: When receiving an access request from a target website, obtaining primary dimension identification information of the access request; Based on all historical access information containing the main dimension identification information within a preset time period, create multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; Generate a feature vector corresponding to the main dimension identification information according to traffic characteristics of multiple single-dimensional conversation sequences and cross-dimensional conversation sequences corresponding to the main dimension identification information; The feature vector corresponding to the main dimension identification information is input into the crawler detection model, and it is determined whether the access request is a crawler request based on the model output content.
14. A network device, characterized in that: The network device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the following method: Obtain historical access information of the target website in the target historical period, and classify the historical access information containing the same main dimension identifier into the same information group; Based on the historical access information under the target information group, multiple single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional session sequence corresponding to the primary dimension identifier are created; wherein the primary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is used to identify different user terminals and distinguish different access requests; the secondary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is in addition to the primary dimension identifier, and is further used to identify different user terminals and distinguish different access requests; Generate a feature vector corresponding to the target information group according to the traffic characteristics corresponding to the cross-dimensional conversation sequence and the traffic characteristics corresponding to the multiple single-dimensional conversation sequences; Based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model.
15. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the following method: Obtain historical access information of the target website in the target historical period, and classify the historical access information containing the same main dimension identifier into the same information group; Based on the historical access information under the target information group, multiple single-dimensional session sequences corresponding to the secondary dimension identifier under the target information group and a cross-dimensional session sequence corresponding to the primary dimension identifier are created; wherein the primary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is used to identify different user terminals and distinguish different access requests; the secondary dimension identifier is an identifier carried in the access request of the user terminal to the target website, and is in addition to the primary dimension identifier, and is further used to identify different user terminals and distinguish different access requests; Generate a feature vector corresponding to the target information group according to the traffic characteristics corresponding to the cross-dimensional conversation sequence and the traffic characteristics corresponding to the multiple single-dimensional conversation sequences; Based on the feature vectors corresponding to all information groups under the target website, a crawler detection model corresponding to the target website is trained, and crawler detection is performed on the target website through the crawler detection model.
