A method, device and storage medium for identifying web crawler behavior

Through the training feature generation of multi-dimensional user behavior data and the training of recognition model, the problem of low accuracy in judging crawler behavior in the prior art is solved, and higher recognition accuracy and wider recognition range are achieved.

CN112989158BActive Publication Date: 2024-11-29SF TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201911290447.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-16
Publication Date
2024-11-29
Estimated Expiration
2039-12-16

AI Technical Summary

Technical Problem

In the prior art, the accuracy rate of judging crawler behavior is not high, and it is easy to misjudge the browsing behavior of normal users as crawler behavior.

Method used

By obtaining user behavior data from multiple dimensions, converting them into training features according to preset rules, generating a sample set and inputting the recognition model for training. Based on the trained recognition model, the behavior data in the access record and the preset behavior characteristics are compared to determine whether the user's access behavior is crawler behavior.

Benefits of technology

It improves the accuracy and comprehensiveness of identifying crawler behavior, reduces misjudgment of normal user behavior, and expands the scope of identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112989158B_ABST
    Figure CN112989158B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a method, device and storage medium for identifying web crawler behavior, the method comprising: obtaining user behavior data of multiple dimensions; converting the user behavior data into a training feature of a preset type according to a preset rule; converting the training feature into a sample set; inputting the sample set into a recognition model to train the recognition model using the sample set; obtaining access records, the access records being at least one behavior data of a user accessing a web page in a historical time; comparing each behavior data in the access record with a preset behavior feature based on the recognition model; if at least one behavior data in the access record matches at least one preset behavior feature, determining that the user's access behavior is a crawler behavior. This solution can improve the comprehensiveness and accuracy of identifying crawler behavior.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular, to a method, device, and storage medium for identifying web crawler behavior. Background Art

[0002] In the prior art, the characteristics of normal users and crawlers accessing web pages are different. For example, normal users will slowly browse a web page by moving the mouse wheel. However, crawlers are different. After a crawler accesses a web page, it will immediately obtain all the information of the web page, transmit the data back to the server, and immediately start crawling the next web page, repeating the cycle. Therefore, the habitual characteristics of normal users and crawlers in accessing web pages are different.

[0003] During the research and practice of the prior art, the inventors of the embodiments of the present application discovered that the accuracy of the prior art in judging crawler behavior is not high, and there is a possibility of misjudging normal users' web browsing behavior as crawler behavior. Summary of the invention

[0004] The embodiments of the present application provide a method, device and storage medium for identifying web crawler behavior, which can improve the comprehensiveness and accuracy of identifying crawler behavior.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying web crawler behavior, the method comprising:

[0006] Obtain user behavior data in multiple dimensions;

[0007] Convert the user behavior data into training features of a preset type according to preset rules;

[0008] Converting the training features into a sample set;

[0009] Inputting the sample set into a recognition model to train the recognition model using the sample set;

[0010] Obtaining access records, wherein the access records are at least one behavior data of a user accessing a web page in a historical period;

[0011] Comparing each behavior data in the access record with preset behavior features based on the recognition model;

[0012] If at least one item of behavior data in the access record matches at least one preset behavior feature, it is determined that the user's access behavior is a crawler behavior.

[0013] In one possible design, the user behavior data includes at least one of the following behavior data:

[0014] The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, whether the page has external links, whether the next page is a jump page to the current page, or whether the page video is clicked to play.

[0015] In a possible design, converting the user behavior data into a preset type of training feature according to a preset rule includes:

[0016] Determine the behavior type of each behavior data in the user behavior data;

[0017] Get the judgment condition that matches the behavior type;

[0018] According to the behavior type, the behavior data matching the behavior type are judged by using the matching judgment conditions respectively to obtain the judgment results of each behavior data;

[0019] According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector;

[0020] The feature vector is used as the training feature.

[0021] In a possible design, converting the training features into a sample set includes:

[0022] According to the preset rules, label the user behavior data of each dimension respectively;

[0023] Associate training features with labels;

[0024] The sample set is generated according to the associated training features and labels.

[0025] In a possible design, each item of behavior data is vectorized according to the judgment result of each item of behavior data to obtain a feature vector, including:

[0026] If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result;

[0027] If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result;

[0028] The flags corresponding to each behavior data are used to form the feature vector.

[0029] In a possible design, after determining that the access behavior of the user is a crawler behavior, the method further includes:

[0030] Generate a verification code, the verification code is used to verify the user's web page access behavior;

[0031] The verification code is sent to the terminal where the user is located.

[0032] In a possible design, after determining that the access behavior of the user is a crawler behavior, the method further includes:

[0033] Obtaining the network address of the terminal where the user is located;

[0034] Block the network address.

[0035] In a second aspect, an embodiment of the present application provides a device for identifying web crawler behavior, which has the function of implementing the method for identifying web crawler behavior provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0036] In one possible design, the device for identifying web crawler behavior includes:

[0037] The transceiver module is used to obtain user behavior data in multiple dimensions;

[0038] A processing module, configured to convert the user behavior data acquired by the transceiver module into a training feature of a preset type according to a preset rule; convert the training feature into a sample set; input the sample set into a recognition model to train the recognition model using the sample set;

[0039] The transceiver module is also used to obtain access records, which are at least one behavior data of a user accessing a web page in a historical period;

[0040] The processing module is also used to compare each behavior data in the access record with preset behavior features based on the recognition model; if at least one behavior data in the access record matches at least one preset behavior feature, it is determined that the user's access behavior is a crawler behavior.

[0041] In one possible design, the user behavior data includes at least one of the following behavior data:

[0042] The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, whether the page has external links, whether the next page is a jump page to the current page, or whether the page video is clicked to play.

[0043] In a possible design, the processing module is specifically used for:

[0044] Determine the behavior type of each behavior data in the user behavior data;

[0045] Get the judgment condition that matches the behavior type;

[0046] According to the behavior type, the behavior data matching the behavior type are judged by using the matching judgment conditions respectively to obtain the judgment results of each behavior data;

[0047] According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector;

[0048] The feature vector is used as the training feature.

[0049] In a possible design, the processing module is specifically used for:

[0050] According to the preset rules, label the user behavior data of each dimension respectively;

[0051] Associate training features with labels;

[0052] The sample set is generated according to the associated training features and labels.

[0053] In a possible design, the processing module is specifically used for:

[0054] If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result;

[0055] If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result;

[0056] The flags corresponding to each behavior data are used to form the feature vector.

[0057] In a possible design, after the processing module determines that the user's access behavior is a crawler behavior, it is further used to:

[0058] Generate a verification code, the verification code is used to verify the user's web page access behavior;

[0059] The transceiver module is provided to send the verification code to the terminal where the user is located.

[0060] In a possible design, after the processing module determines that the user's access behavior is a crawler behavior, it is further used to:

[0061] Obtaining the network address of the terminal where the user is located;

[0062] Block the network address.

[0063] On the other hand, an embodiment of the present application provides an apparatus for identifying web crawler behavior, which includes at least one connected processor, memory and transceiver, wherein the memory is used to store a computer program, and the processor is used to call the computer program in the memory to execute the method described in the first aspect above.

[0064] Another aspect of the embodiments of the present application provides a computer-readable storage medium, which includes instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect above.

[0065] Compared with the prior art, the solution provided in the embodiment of the present application, on the one hand, trains the recognition model from multiple dimensions, so that the recognition accuracy of the recognition model is higher, thereby improving the comprehensive evaluation of whether an access request is a crawler behavior and reducing misjudgment. On the other hand, the input of the recognition model in the embodiment of the present application is a training feature of multiple dimensions, so it can cover crawler behaviors in various situations, and will not miss other features that can be used to judge whether an access request is a crawler behavior, so the recognition range is wider. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of a method for identifying web crawler behavior in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a structure of an identification model in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the structure of a device for identifying web crawler behavior in an embodiment of the present application;

[0069] Figure 4 A structural schematic diagram of a computer device that executes the method for identifying web crawler behavior in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of the modules that appear in the embodiments of the present application is only a logical division. There may be other division modes when implemented in practical applications, such as multiple modules can be combined into or integrated in another system, or some features can be ignored, or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in the embodiments of the present application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0071] The embodiments of the present application provide a method, device and storage medium for identifying web crawler behavior, which can be used on the server side of a crawler system. The server side can be used to identify whether the access behavior of a web page is crawler behavior.

[0072] The embodiments of the present application mainly provide the following technical solutions:

[0073] Improve the input of the recognition model, that is, use user behavior data in multiple dimensions to train the recognition model, so that the recognition model can identify whether the web page access behavior is crawler behavior from multiple dimensions, and can comprehensively evaluate whether the web page access behavior is crawler behavior, with better accuracy and a wide recognition range.

[0074] Reference Figure 1 , the following describes a method for identifying web crawler behavior provided by an embodiment of the present application. The execution subject of the method may be a server, or a device for identifying web crawler behavior. The device for identifying web crawler behavior may be a server or an application installed on a server, and the embodiment of the present application does not limit this. The embodiment of the present application includes:

[0075] 101. Obtain user behavior data in multiple dimensions.

[0076] Among them, the user behavior data can be the historical data of users visiting the website, or it can be the operation log saved in the server background. The embodiment of the present application does not limit the acquisition channel of the user behavior data, whether to perform preprocessing, etc.

[0077] In some implementations, the user behavior data includes at least one of the following behavior data:

[0078] The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, the existence of external links on the page, and whether the next page is a jump page to the current page.

[0079] The webpage dwell time refers to the time a user stays on a page, which can also be called the webpage dwell time. Generally, a user will stay on a webpage for a while. If the dwell time is very short, it may be a crawler. Therefore, a threshold can be defined for the webpage dwell time. If the time difference between entering and leaving the webpage is greater than the threshold, the value of the webpage dwell time feature is 1, otherwise it is 0. For example, the threshold of the webpage dwell time is set to 1 minute.

[0080] The number of times the page wheel is scrolled, including the number of times the wheel is scrolled down and the number of times the wheel is scrolled up. Specifically, the number of times the wheel is scrolled down: After opening a web page, a normal user will use the wheel to slide up and down to view the content of the web page, but general crawlers do not use the wheel (not absolutely, some crawlers simulate clicks). Therefore, it is easy to determine whether it is crawler behavior by the number of times the page wheel is scrolled, so a threshold can be defined. If the number of times the wheel is scrolled down is greater than the threshold, the value of this feature is 1, otherwise it is 0. The number of times the wheel is scrolled up: Even if the crawler may use the wheel by simulating clicks, it rarely scrolls up, so a threshold can be defined for the number of times the page wheel is scrolled. If the number of times the wheel is scrolled up is greater than the threshold, the value of the number of times the page wheel is scrolled is 1, otherwise it is 0.

[0081] The scrolling time interval of the page wheel includes the time interval of the wheel scrolling up and the time interval of the wheel scrolling down. Or the scrolling time interval of the page wheel is called the scrolling frequency of the page wheel, and the scrolling frequency of the page wheel includes the frequency of the wheel scrolling up or down. Normally, normal users read a page and then scroll up or down. It takes a certain amount of time to read a page, so generally speaking, crawlers will not randomly scroll pages on a page. Therefore, the scrolling time interval of the page wheel can be used to judge crawler behavior.

[0082] The number of click events on the page, such as the number of mouse clicks on the page or user gesture input. When the number of click events on the page is the number of mouse clicks on the page, since many normal users are accustomed to double-clicking the web page with the left button of the mouse when visiting the web page, a threshold can be defined. If the number of double-clicks on the web page with the left button is greater than the threshold, the value of the feature of the number of click events on the page is 1, otherwise it is 0.

[0083] The time interval between adjacent click events, such as the time interval between two or more mouse clicks. When the time interval between adjacent click events is the time interval between two mouse clicks, since crawlers generally do not simulate mouse clicks (unless necessary steps, such as web page 2 must be entered through a button on web page 1), a threshold can be defined for the time interval between adjacent click events. If the average click interval is greater than the threshold, the value of the time interval between adjacent click events is 1, otherwise it is 0.

[0084] The page has external links, such as whether the external links are clicked: a page may have some links that are not on this website (external links). Crawlers generally do not visit external links, only pages under the same domain name. Therefore, if there is a click on the external link, the page has external links. This feature is 1, otherwise it is 0.

[0085] Whether the next page is a jump from this page. Normal users browse the web and usually jump through the current page, but crawlers may not necessarily jump through the current page. If it is a breadth search crawler, it basically does not jump from the current page. Therefore, if the next page is a jump from the current page, the value of this feature is 1, otherwise it is 0.

[0086] Whether the page video has been clicked to play, for example, whether the video in the site has been clicked. Since ordinary users may click on the videos in the site, crawlers basically will not click on the videos in the site or cannot click on the videos in the site. Therefore, if it is clicked, the feature value of whether the page video has been clicked to play is 1, otherwise it is 0.

[0087] 102. Convert the user behavior data into a preset type of training features according to preset rules.

[0088] Among them, the preset rules include: dividing the features of different dimensions into different preset types, and defining the quantification method of the features of the preset types according to different preset types. For example, the user behavior data includes: user A visited web page 1 from 2019-12-1 08:00:05 to 2019-12-1 08:02:21, and visited web page 2 from 2019-12-1 08:04:45 to 2019-12-1 08:12:34. Such data cannot be used directly to train the recognition model, so the user behavior data is quantified as user A's residence time on web page 1 is 2 minutes and 16 seconds, and the residence time on web page 2 is 7 minutes and 49 seconds. The same is true for the others, so I will not go into details.

[0089] In some implementations, converting the user behavior data into a preset type of training feature according to a preset rule includes:

[0090] a. Determine the behavior type of each behavior data in the user behavior data.

[0091] b. Obtain the judgment condition that matches the behavior type.

[0092] c. According to the behavior type, the behavior data matching the behavior type are judged using the matching judgment conditions to obtain the judgment results of each behavior data.

[0093] d. According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector.

[0094] In some implementations, the vectorization processing is performed on each behavior data according to the judgment result of each behavior data to obtain a feature vector, including:

[0095] If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result;

[0096] If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result;

[0097] The flags corresponding to each behavior data are used to form the feature vector.

[0098] Among them, the first mark and the second mark can be numbers, symbols or text, or a combination of at least one of numbers, symbols or text. The embodiment of the present application does not limit the expression of the first mark and the second mark, as long as it can distinguish the judgment results of each behavior data and does not affect the formation of subsequent feature vectors.

[0099] For example, if the judgment result is: the web page dwell time is 2 minutes, the page wheel scrolling times is 4, and the next page is the jump page of the current page, then these are all positive judgment results. Therefore, the first flags can be set for the behavior data of these items respectively, for example, all set to 1.

[0100] For another example, if the web page dwell time is 0.01 seconds, the number of page scrolling is 2, the next page is not the jump page of the current page, and the page video is not clicked to play, then these are all negative judgment results. Therefore, the second flags can be set for the behavior data of these items respectively, for example, the second flags are all set to 0.

[0101] e. Using the feature vector as the training feature.

[0102] For example, user behavior data includes:

[0103] The web page residence time is 2 minutes, the page wheel scrolls 4 times, the page wheel scrolls for 4 seconds, the page click events number is 2, the time interval between adjacent click events is 1 second, the page has external links, the next page is the jump page of the current page, or the page video is clicked to play.

[0104] Therefore, the training features can be expressed as:

[0105] [1,1,1,1,1,1,1,1]

[0106] For another example, the web page dwell time is 0.01 seconds, the page wheel scrolls 2 times, the page wheel scrolling time interval is 4 seconds, the number of click events on the page is 0, the time interval between adjacent click events is 0 seconds, the page has external links, the next page is not a jump page for the current page, or the page video has not been clicked to play.

[0107] Therefore, the training features can be expressed as:

[0108] [0,0,1,0,0,1,0,0]

[0109] Among them, 1 indicates that the user meets the judgment condition of a certain training feature, and 0 indicates that the user does not have the above training feature or does not meet the judgment condition of a certain training feature. The content structure can be an ordered sequence or an array, which is not limited in the embodiment of the present application.

[0110] 103. Convert the training features into a sample set.

[0111] Here, the sample set refers to the feature set used to train the recognition model.

[0112] In some implementations, converting the training features into a sample set includes:

[0113] According to the preset rules, label the user behavior data of each dimension respectively;

[0114] Associate training features with labels;

[0115] The sample set is generated according to the associated training features and labels.

[0116] Normal user access records and crawler access records. Among them, crawler records can be manually marked by website administrators, or obtained through anti-crawler methods. Anti-crawler methods can be obtained by detecting network addresses, user agents (UA) detection, etc. The embodiments of this application do not limit the acquisition method and acquisition channel of crawler access records.

[0117] For example, after setting corresponding labels for user behavior features, the labels are combined with the above training features to form a sample set:

[0118] [1,1,1,1,1,1,1,1] [0]

[0119] [0,0,1,0,0,1,0,0] [1]

[0120] Among them, [1,1,1,1,1,1,1,1] and [0,0,1,0,0,1,0,0] are both training features, and [0] and [1] are both labels. [0] is the label of the training feature [1,1,1,1,1,1,1,1], and [1] is the label of the training feature [0,0,1,0,0,1,0,0].

[0121] 104. Input the sample set into a recognition model to train the recognition model using the sample set.

[0122] Among them, the recognition model can be used to identify whether the web page access behavior is crawler behavior or normal user access behavior.

[0123] In some implementations, the recognition model can be obtained by training tree models such as xgboost, random forest, support vector machine (Support Vector Machine, svm), decision tree, etc., so that the recognition model obtained has higher accuracy. When training the above recognition model, in order to be close to the actual business, the hyperparameters of the recognition model can be continuously adjusted to gradually find an optimal recognition model suitable for this business.

[0124] The following describes the training process of the recognition model:

[0125] The recognition model includes multiple hyperparameters, and the recognition model can be obtained by training using a variety of machine learning methods. For example, the recognition model can be obtained by training using xgboost or svm.

[0126] Among the hyperparameters of the recognition model, the learning rate and the depth of the tree have a relatively large impact on the recognition accuracy of the recognition model, so the embodiment of the present application mainly takes the adjustment of the learning rate and the depth of the tree as an example. For example, when the recognition model is obtained by xgboost training, the learning rate of xgboost and the depth of the tree can be continuously adjusted. For example, the last adjusted learning rate is 0.15, and the depth of the tree is 4, and the recognition effect of the recognition model is the best. Therefore, the learning rate of the recognition model can be set to a value range of [0.1, 0.2] by default, for example, the default learning rate of xgboost is 0.15. And the time step of the recognition model is set to a value range of, for example, if the time step is 0.05, then the learning rate can be selected from [0.1, 0.15, 0.2]. For another example, the depth of the tree is 3 by default, and the limited range is [2, 6]. If the time step is 1, then this parameter can be selected from [2, 3, 4, 5, 6]. The values ​​of other parameters of the recognition model can be deduced in turn. Specifically, adjusting the hyperparameters of the recognition model is to match different optional hyperparameters, try all possibilities, and then find an optimal solution.

[0127] For example, when the training features include dwell time and scroll times, the existing features of the user (i.e., dwell time and scroll times) are considered to build a recognition model, so that the recognition model at the training stage can identify whether the web page access behavior is crawler behavior based on these two types of features. The quality of the recognition model can also be evaluated based on the existing labels, that is, the higher the accuracy of the predicted labels, the better the recognition effect of the recognition model.

[0128] In some implementations, in order to improve the recognition effect of the recognition model, the recognition model can be used as follows: Figure 2 The model structure shown. The recognition model can use a three-layer random forest network, and each layer of the network can process each training feature separately. For example, a three-layer network can be used to train one or more training features separately, so as to achieve a better recognition effect, and the output of each layer of the network is summarized and weighted, and the weighted result (i.e., hn) is used to determine whether the training features input into the entire recognition model this time are consistent with the normal user access behavior or crawler behavior. Figure 3 As shown, the access records of the input recognition model are Fn, Fn-1, Fn-2, ... Fn-m+1, and the weighted result is hn. After hn is mapped to 2 layers of fully connected layers (dense layers), the recognition result Yn is output.

[0129] 105. Get access records.

[0130] The access record is at least one item of behavior data of a user accessing a web page in a historical period. For example, the access record may include at least one of the following items:

[0131] The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, whether the page has external links, whether the next page is a jump page to the current page, or whether the page video is clicked to play.

[0132] 106. Compare each behavior data in the access record with a preset behavior feature based on the recognition model; if at least one behavior data in the access record matches at least one preset behavior feature, determine that the user's access behavior is a crawler behavior.

[0133] For example, if a user (possibly a crawler) is accessing a web page, then the behavior log of the accessed web page within a period of time is obtained, and then the recognition model is used to determine whether the behavior log model within this period of time is a crawler behavior. If it is a crawler, the request to access the web page is directly disconnected to stop the current operation of accessing the web page; or a verification request is sent to the user on the web page, and the verification request is used to request to enter a verification code on the web page.

[0134] In some embodiments, based on Figure 2 The structure of the recognition model shown in the figure judges each behavior data in the access record. Finally, the output of each layer is summarized and weighted to obtain the weighted result. According to the weighted result, it is judged whether the overall training features of the input recognition model are consistent with user access or crawler access.

[0135] In some implementations, after determining that the user's access behavior is a crawler behavior, the recognition model will output a recognition result, which is used to indicate that the current user's access behavior is a crawler behavior. It is also possible to decide to perform some operations on the user currently accessing the web page based on the recognition result of the recognition model. Specifically, it includes the following two aspects:

[0136] (1) Use verification codes to prevent further network access

[0137] In some implementations, a verification code may be generated, the verification code being used to verify the user's web page access behavior, and the verification code may be sent to a terminal where the user is located.

[0138] The verification code may be generated according to a rule or randomly, and the embodiment of the present application does not limit the generation method of the verification code.

[0139] Optionally, a timer can be set. If the verification code entered by the user in the verification code interface is not received within a preset time, a timeout verification is directly returned to the user's terminal. Alternatively, if the verification code entered by the user in the verification code interface is incorrect, a verification failure is directly returned to the user's terminal.

[0140] (2) Blocking further network access by blocking network addresses

[0141] In some implementations, the network address of the terminal where the user is located can be obtained and the network address can be blocked. For example, when the network address is an Internet protocol (IP) address, the IP address or the IP segment of the IP address can be blocked. The embodiment of the present application does not limit the method and content of blocking the network address. The account logged in by the user or the terminal identifier of the terminal where the user is located can also be blocked, which is not limited in the embodiment of the present application.

[0142] In the embodiment of the present application, the training features used for training the model are comprehensively considered. The training features can comprehensively analyze the suspicious features related to crawler behavior from various dimensions (i.e., the time spent on the web page, the number of times the wheel is scrolled down, the number of times the wheel is scrolled up, the number of mouse clicks on the page, the time interval between two mouse clicks, whether to click on an external link, whether the next page is a jump to the current page, whether to click on a video on the site, and other user behavior data). On the one hand, by training the model from multiple dimensions, the recognition accuracy of the model is higher, so it can improve the comprehensive evaluation of whether an access request is a crawler behavior and reduce misjudgment. On the other hand, the input of the model in the present invention is a variety of dimensional suspicious features, so it can cover crawler behavior in a variety of situations, and will not miss other features that can be used to judge whether an access request is a crawler behavior, so the recognition range is wider.

[0143] Figure 1 , Figure 2 Any technical feature mentioned in any embodiment corresponding to any one of the items is also applicable to the embodiments of the present application. Figure 3 and Figure 4 The corresponding embodiments will not be described in detail later.

[0144] A method for identifying web crawler behavior in an embodiment of the present application is described above. The following describes an apparatus for identifying web crawler behavior in an embodiment of the present application.

[0145] See also Figure 3 ,like Figure 3 The schematic diagram of the structure of a device 30 for identifying web crawler behavior is shown in FIG. 3 , which can be applied to identifying crawler behavior. The device 30 for identifying web crawler behavior in the embodiment of the present application can achieve the corresponding Figure 1The steps of the method for identifying web crawler behavior performed in the corresponding embodiment. The functions implemented by the device 30 for identifying web crawler behavior can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The device 30 for identifying web crawler behavior may include a processing module and a transceiver module. The functions of the processing module and the transceiver module can be implemented by reference Figure 1 The operations performed in the corresponding embodiments are not described in detail here. For example, the processing module can be used to control the acquisition, receiving and sending operations of the receiving and sending module.

[0146] In some implementations, the transceiver module may be used to obtain user behavior data in multiple dimensions;

[0147] The processing module can be used to convert the user behavior data obtained by the transceiver module into a preset type of training features according to preset rules; convert the training features into a sample set; input the sample set into the recognition model to train the recognition model using the sample set;

[0148] The transceiver module is also used to obtain access records, which are at least one behavior data of a user accessing a web page in a historical period;

[0149] The processing module is also used to compare each behavior data in the access record with preset behavior features based on the recognition model; if at least one behavior data in the access record matches at least one preset behavior feature, it is determined that the user's access behavior is a crawler behavior.

[0150] In the embodiment of the present application, on the one hand, the recognition model is trained from multiple dimensions, so that the recognition accuracy of the recognition model is higher, thereby improving the comprehensive evaluation of whether an access request is a crawler behavior and reducing misjudgment. On the other hand, the input of the recognition model in the embodiment of the present application is a multi-dimensional training feature, which is obtained based on user behavior data of multiple dimensions, so it can cover crawler behaviors in various situations, and will not miss other features that can be used to judge whether an access request is a crawler behavior, so the recognition range is wider.

[0151] In some implementations, the user behavior data includes at least one of the following behavior data:

[0152] The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, whether the page has external links, whether the next page is a jump page to the current page, or whether the page video is clicked to play.

[0153] In some implementations, the processing module is specifically used to:

[0154] Determine the behavior type of each behavior data in the user behavior data;

[0155] Get the judgment condition that matches the behavior type;

[0156] According to the behavior type, the behavior data matching the behavior type are judged by using the matching judgment conditions respectively to obtain the judgment results of each behavior data;

[0157] According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector;

[0158] The feature vector is used as the training feature.

[0159] In some implementations, the processing module is specifically used to:

[0160] According to the preset rules, label the user behavior data of each dimension respectively;

[0161] Associate training features with labels;

[0162] The sample set is generated according to the associated training features and labels.

[0163] In some implementations, the processing module is specifically used to:

[0164] If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result;

[0165] If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result;

[0166] The flags corresponding to each behavior data are used to form the feature vector.

[0167] In some implementation modes, after determining that the user's access behavior is a crawler behavior, the processing module is further configured to:

[0168] Generate a verification code, the verification code is used to verify the user's web page access behavior;

[0169] The transceiver module is provided to send the verification code to the terminal where the user is located.

[0170] In some implementation modes, after determining that the user's access behavior is a crawler behavior, the processing module is further configured to:

[0171] Obtaining the network address of the terminal where the user is located;

[0172] Block the network address.

[0173] The network authentication server and the terminal device in the embodiment of the present application are described above from the perspective of modular functional entities. The network authentication server and the terminal device in the embodiment of the present application are described below from the perspective of hardware processing.

[0174] It should be noted that in the embodiments of this application Figure 3 The physical device corresponding to the transceiver module in the illustrated embodiment may be a transceiver, and the physical device corresponding to the processing module may be a processor. Figure 3 The devices 30 shown may have Figure 4 As shown in the structure, when the device 30 has Figure 4 When the structure shown is Figure 4 The processor and transceiver in the device implement the same or similar functions as the processing module and the transceiver module provided in the device embodiment corresponding to the device, Figure 4 The memory in the storage device stores a computer program that needs to be called when the processor executes the above-mentioned method for identifying web crawler behavior.

[0175] For example, the processor calls a computer program in memory to perform the following operations:

[0176] Acquire user behavior data of multiple dimensions through the transceiver;

[0177] Convert the user behavior data acquired by the transceiver module into a preset type of training features according to preset rules; convert the training features into a sample set; input the sample set into a recognition model to train the recognition model using the sample set;

[0178] Acquire access records through the transceiver, wherein the access records are at least one behavior data of a user accessing a web page in a historical period;

[0179] Based on the recognition model, each behavior data in the access record is compared with the preset behavior characteristics respectively; if at least one behavior data in the access record matches at least one preset behavior characteristic, it is determined that the user's access behavior is a crawler behavior.

[0180] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0183] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0185] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0186] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0187] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A method for identifying web crawler behavior, characterized in that: The method comprises: Obtain user behavior data in multiple dimensions; Convert the user behavior data into training features of a preset type according to preset rules; Converting the training features into a sample set; Inputting the sample set into a recognition model to train the recognition model using the sample set; Obtaining access records, wherein the access records are at least one behavior data of a user accessing a web page in a historical period; Comparing each behavior data in the access record with preset behavior features based on the recognition model; If at least one item of behavior data in the access record matches at least one preset behavior feature, it is determined that the user's access behavior is a crawler behavior; The converting of the user behavior data into a preset type of training feature according to a preset rule includes: Determine the behavior type of each behavior data in the user behavior data; Get the judgment condition that matches the behavior type; According to the behavior type, the behavior data matching the behavior type are judged by using the matching judgment conditions respectively to obtain the judgment results of each behavior data; According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector; Using the feature vector as the training feature; The vectorization process is performed on each behavior data according to the judgment result of each behavior data to obtain a feature vector, including: If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result; If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result; The flags corresponding to each behavior data form the feature vector; After determining that the user's access behavior is a crawler behavior, an operation is performed on the user currently accessing the webpage based on the recognition result of the recognition model, specifically including: Using a verification code to prevent network access behavior, specifically: generating a verification code, the verification code is used to verify the user's web page access behavior; sending the verification code to the terminal where the user is located; if the verification code entered by the user in the verification code interface is not received within a preset time, directly returning a timeout verification to the terminal where the user is located; or, if the verification code entered by the user in the verification code interface is incorrect, directly returning a verification failure to the terminal where the user is located; or, The network access behavior is prevented by blocking the network address, specifically: obtaining the network address of the terminal where the user is located, the account the user is logged in to, or the terminal identification of the terminal where the user is located; blocking the network address, blocking the account, or blocking the terminal identification; the network address blocking is to block the IP address or block the IP segment of the IP address.

2. The method according to claim 1, characterized in that The user behavior data includes at least one of the following behavior data: The length of time a web page is retained, the number of times the page wheel is scrolled, the scrolling time interval of the page wheel, the number of click events on the page, the time interval between adjacent click events, whether the page has external links, whether the next page is a jump page to the current page, or whether the page video is clicked to play.

3. The method according to claim 1, characterized in that The converting the training features into a sample set comprises: According to the preset rules, label the user behavior data of each dimension respectively; Associate training features with labels; The sample set is generated according to the associated training features and labels.

4. A device for identifying web crawler behavior, characterized in that: The device comprises: The transceiver module is used to obtain user behavior data in multiple dimensions; A processing module, configured to convert the user behavior data acquired by the transceiver module into a training feature of a preset type according to a preset rule; convert the training feature into a sample set; input the sample set into a recognition model to train the recognition model using the sample set; The transceiver module is also used to obtain access records, which are at least one behavior data of a user accessing a web page in a historical period; The processing module is also used to compare each behavior data in the access record with the preset behavior characteristics based on the recognition model; if at least one behavior data in the access record matches at least one preset behavior characteristic, it is determined that the user's access behavior is a crawler behavior; The converting of the user behavior data into a preset type of training feature according to a preset rule includes: Determine the behavior type of each behavior data in the user behavior data; Get the judgment condition that matches the behavior type; According to the behavior type, the behavior data matching the behavior type are judged by using the matching judgment conditions respectively to obtain the judgment results of each behavior data; According to the judgment results of each behavior data, each behavior data is vectorized to obtain a feature vector; Using the feature vector as the training feature; The vectorization process is performed on each behavior data according to the judgment result of each behavior data to obtain a feature vector, including: If the judgment result is positive, a first flag is set for the behavior data corresponding to the positive judgment result; If the judgment result is negative, a second flag is set for the behavior data corresponding to the negative judgment result; The flags corresponding to each behavior data form the feature vector; After determining that the user's access behavior is a crawler behavior, an operation is performed on the user currently accessing the webpage based on the recognition result of the recognition model, specifically including: Using a verification code to prevent network access behavior, specifically: generating a verification code, the verification code is used to verify the user's web page access behavior; sending the verification code to the terminal where the user is located; if the verification code entered by the user in the verification code interface is not received within a preset time, directly returning a timeout verification to the terminal where the user is located; or, if the verification code entered by the user in the verification code interface is incorrect, directly returning a verification failure to the terminal where the user is located; or, The network access behavior is prevented by blocking the network address, specifically: obtaining the network address of the terminal where the user is located, the account the user is logged in to, or the terminal identification of the terminal where the user is located; blocking the network address, blocking the account, or blocking the terminal identification; the network address blocking is to block the IP address or block the IP segment of the IP address.

5. A computer device, characterized in that: The computer device comprises: at least one processor, memory, and transceiver; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Method and device for identifying web crawler and extracting web crawler features

    CN109145185A

  • A crawler identification method based on user mouse interaction behavior

    CN109189660A