A crawler behavior detection method, device, equipment and storage medium
By combining sparse nonnegative matrix factorization and BiLSTM+Attention network, the accuracy problem of existing web crawler detection methods under unlabeled data is solved, and efficient and accurate web crawler behavior recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG AISINO CO LTD
- Filing Date
- 2022-12-07
- Publication Date
- 2026-06-09
Smart Images
Figure CN115730112B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data leakage prevention technology, and in particular to a method, apparatus, equipment and storage medium for detecting web crawler behavior. Background Technology
[0002] Web crawlers, also known as web crawler robots, are programs or scripts that automatically retrieve information from the internet according to certain rules. With the rapid development of internet technology, the internet carries a vast amount of valuable information. Web crawlers satisfy many people's needs for information and data by scraping from this massive amount of data. However, web crawling also poses security risks. For example, unauthorized data providers using topic-based crawlers to indiscriminately scrape large amounts of website data can impact server performance; illegally collecting website account passwords and other security information can cause economic losses to businesses; and bulk access by crawlers can also increase server load, affecting access for legitimate users.
[0003] Currently, relevant web crawler detection methods can be mainly divided into three categories: syntax rule-based detection, traffic pattern-based detection, and behavior pattern-based detection. Syntax rule-based detection methods identify web crawlers by checking whether access logs contain keywords, whether the User-Agent contains a crawler identifier, and whether the host IP address matches a crawler address database. Traffic pattern-based detection methods, building upon syntax rule-based methods, consider the crawler's traffic patterns within a session. However, crawlers can evade syntax rule detection by modifying field information or using long-term access cycles to avoid exhibiting crawler traffic patterns. Based on traffic pattern detection, behavior pattern-based detection methods have been proposed, considering the differences between crawlers and normal users in traversal patterns or topic content access patterns. However, all three types of detection methods are specific to certain websites or services and require significant manual analysis and intervention. Their detection performance inevitably declines when faced with large amounts of unlabeled data. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for detecting web crawler behavior. This method can filter access behavior data through sparse non-negative matrix factorization, solving the problem of large access traffic and lack of data labels. Furthermore, it can utilize a pre-set BiLSTM+Attention network to improve the accuracy of the final result. The specific solution is as follows:
[0005] Firstly, this application provides a method for detecting web crawler behavior, including:
[0006] Obtain raw data on user access behavior;
[0007] The original data is processed according to preset data processing rules to obtain standard data, and the standard data is vectorized according to preset vectorization rules to obtain corresponding behavioral feature vectors.
[0008] The similarity matrix corresponding to the behavior feature vector is normalized to obtain the corresponding normalized similarity matrix, and the normalized similarity matrix is subjected to sparse non-negative matrix decomposition to obtain the initial crawler behavior screening results corresponding to the original data.
[0009] The initial crawler behavior screening results are predicted using a pre-defined BiLSTM+Attention network to obtain the crawler behavior prediction results.
[0010] Optionally, the step of processing the raw data according to preset data processing rules to obtain standard data includes:
[0011] The original data is preprocessed to obtain preprocessed data;
[0012] The feature information of the preprocessed data is extracted to obtain the standard data; wherein the standard data includes: username, user IP address, location corresponding to the IP address, user behavior, and behavior time.
[0013] Optionally, the step of vectorizing the standard data based on preset vectorization rules to obtain the corresponding behavioral feature vectors includes:
[0014] The user behavior and the behavior time in the standard data are vectorized based on one-hot encoding to obtain the corresponding behavior vector and time weight vector.
[0015] The behavior vector and the time weight vector are concatenated and fused to obtain the access behavior vector, and the access behavior vector and a preset number of user behavior information before and after the corresponding user behavior are concatenated to obtain the behavior feature vector.
[0016] Optionally, the step of normalizing the similarity matrix corresponding to the behavior feature vector to obtain a corresponding normalized similarity matrix, and performing sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior screening results corresponding to the original data, includes:
[0017] Calculate the similarity matrix between the behavioral feature vectors, and normalize the similarity matrix to obtain the normalized similarity matrix;
[0018] The initial crawler behavior screening results are obtained by performing sparse non-negative matrix decomposition on the normalized similarity matrix according to a preset matrix decomposition function; wherein, the preset matrix decomposition function includes a preset risk level and a preset clustering indicator matrix.
[0019] Optionally, the step of using a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results includes:
[0020] The initial prediction result is obtained by using the normalized exponential function to predict the initial crawler behavior screening result in the preset BiLSTM+Attention network.
[0021] The category with the highest score in the initial prediction results is determined as the crawler behavior prediction result.
[0022] Optionally, before using a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain the crawler behavior prediction results, the method further includes:
[0023] The BiLSTM+Attention network is trained using the behavioral feature vector to obtain the preset BiLSTM+Attention network.
[0024] Optionally, after obtaining the initial crawler behavior filtering results corresponding to the original data, the method further includes:
[0025] The crawler behavior data and non-crawler behavior data in the initial crawler behavior screening results are respectively identified as positive samples and negative samples;
[0026] The preset BiLSTM+Attention network is trained and optimized using the positive and negative samples, and the optimized BiLSTM+Attention network is determined as the preset BiLSTM+Attention network for the next use.
[0027] Optionally, after predicting the initial crawler behavior screening results using a preset BiLSTM+Attention network to obtain the crawler behavior prediction results, the method further includes:
[0028] The crawler behavior data in the initial crawler behavior screening results is determined as test data;
[0029] The optimized BiLSTM+Attention network is validated using the test data to obtain the corresponding evaluation results.
[0030] Secondly, this application provides a web crawler behavior detection device, comprising:
[0031] The data acquisition module is used to acquire raw data on user access behavior;
[0032] The data vectorization module is used to process the original data according to preset data processing rules to obtain standard data, and to convert the standard data into corresponding behavioral feature vectors based on preset vectorization rules.
[0033] The filtering result determination module is used to normalize the similarity matrix corresponding to the behavior feature vector to obtain the corresponding normalized similarity matrix, and to perform sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior filtering result corresponding to the original data.
[0034] The prediction result determination module is used to predict the crawler behavior prediction result by using a preset BiLSTM+Attention network to predict the initial crawler behavior screening result.
[0035] Thirdly, this application provides an electronic device, comprising:
[0036] Memory, used to store computer programs;
[0037] A processor is used to execute the computer program to implement the steps of the above-described crawler behavior detection method.
[0038] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the above-described crawler behavior detection method.
[0039] Therefore, this application can obtain raw data of user access behavior; then, it processes the raw data according to preset data processing rules to obtain standard data, and performs vectorization processing on the standard data based on preset vectorization rules to obtain corresponding behavior feature vectors; then, it normalizes the similarity matrix corresponding to the behavior feature vectors to obtain the corresponding normalized similarity matrix, and performs sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior screening results corresponding to the raw data; then, it uses a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results. In this way, this application can vectorize standard data to obtain behavior feature vectors, and can extract in-depth and comprehensive access behavior information; and it can perform preliminary screening of the behavior feature vectors corresponding to the raw data through sparse non-negative matrix decomposition, which can solve the problem of large access traffic and no data labels, while obtaining accurate results; then, it can perform secondary detection through the preset BiLSTM+Attention network, which can solve the problem of easy misjudgment, thereby improving the accuracy of crawler behavior detection. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0041] Figure 1 This is a flowchart of a web crawler behavior detection method disclosed in this application;
[0042] Figure 2 This is a flowchart of a specific web crawler behavior detection method disclosed in this application;
[0043] Figure 3 This is a flowchart of a data vectorization process disclosed in this application;
[0044] Figure 4 This is a flowchart of a specific web crawler behavior detection method disclosed in this application;
[0045] Figure 5 This is a flowchart of a specific web crawler behavior detection method disclosed in this application;
[0046] Figure 6 This is a flowchart of a specific web crawler behavior detection method disclosed in this application;
[0047] Figure 7 This is a schematic diagram of the structure of a crawler behavior detection device disclosed in this application;
[0048] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0050] Existing methods for detecting web crawler behavior are all targeted at specific websites or services and involve significant manual analysis and intervention. Therefore, this application proposes a highly reusable and accurate method for detecting web crawler behavior. Combining unsupervised and supervised methods can effectively utilize large amounts of unlabeled data while achieving high detection accuracy. Sparse Nonnegative Matrix Factorization (SNMF) is an efficient, fast, and always-solvable unsupervised method. BiLSTM (Bi-directional Long Short-Term Memory) is a deep network model that can effectively identify contextual sequence features, while the Attention model can identify important information across different domains.
[0051] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a method for detecting crawler behavior, including:
[0052] Step S11: Obtain raw data of user access behavior.
[0053] In this embodiment, the original access behavior data related to user access behavior can be obtained first. It is understood that there are certain behavioral differences between crawlers and normal users in traversal mode or topic content access mode. This application can process some access behavior data according to the behavioral differences to determine whether there is data related to crawler behavior, so as to ensure information security in the network.
[0054] Step S12: Process the original data according to the preset data processing rules to obtain standard data, and perform vectorization processing on the standard data according to the preset vectorization rules to obtain the corresponding behavioral feature vector.
[0055] In this embodiment, after obtaining the raw data of user access behavior, the raw data can be processed to obtain standard data. Then, the obtained standard data can be vectorized using the preset vectorization rule to obtain the behavior feature vector corresponding to the standard data. It should be noted that, in a specific embodiment, after obtaining the behavior feature vector, the process may include: training a BiLSTM+Attention network using the behavior feature vector to obtain the preset BiLSTM+Attention network. Further, in a specific embodiment, when training the BiLSTM+Attention network, the batch size parameter can be set to 16, and then 90% of all behavior feature vectors obtained in step S12 can be used as the training set, and 10% as the test set. It is understood that BiLSTM is a deep network model that can effectively identify contextual sequence features, while the Attention model can identify important information across domains. This application can use the behavior feature vector obtained from the standard data to train the BiLSTM+Attention network to obtain the preset BiLSTM+Attention network.
[0056] Step S13: Normalize the similarity matrix corresponding to the behavior feature vector to obtain the corresponding normalized similarity matrix, and perform sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior screening results corresponding to the original data.
[0057] In this embodiment, after obtaining the behavior feature vectors, a similarity matrix between the behavior feature vectors can be calculated. Then, the similarity matrix can be normalized to obtain a normalized similarity matrix. Finally, sparse non-negative matrix factorization of the obtained normalized similarity matrix yields the partitioning result corresponding to the behavior feature vectors, i.e., the initial crawler behavior screening result. It should be noted that this application can utilize the initial crawler behavior screening result to optimize a preset BiLSTM+Attention network. This can include: determining crawler behavior data and non-crawler behavior data in the initial crawler behavior screening result as positive and negative samples, respectively; using the positive and negative samples to train and optimize the preset BiLSTM+Attention network; and determining the optimized BiLSTM+Attention network as the preset BiLSTM+Attention network for the next use. In this way, this application can optimize the detection network used during crawler behavior data detection, allowing it to be reused in the next crawler behavior detection task, thus continuously optimizing the accuracy of the detection network in crawler behavior detection.
[0058] Step S14: Use a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results.
[0059] In this embodiment, after obtaining the initial crawler behavior screening results, the preset BiLSTM+Attention network can be used to predict the crawler behavior prediction results from the initial crawler behavior screening results. It should be noted that this application can evaluate the performance of the detection network, which may include: determining the crawler behavior data in the initial crawler behavior screening results as test data; and using the test data to validate the optimized BiLSTM+Attention network to obtain the corresponding evaluation results. Specifically, the evaluation metrics may include precision, recall, and the overall evaluation metric F1, calculated as follows:
[0060]
[0061]
[0062]
[0063] In this context, TP indicates that a positive sample is predicted as a positive sample, FP indicates that a negative sample is predicted as a positive sample, and FN indicates that a positive sample is predicted as a negative sample. This allows for accurate assessment of the detection accuracy of the detection network used during the crawler behavior detection process.
[0064] Therefore, this application can construct a detection network based on BiLSTM+Attention using a small amount of training data, and the BiLSTM+Attention network can be trained and optimized during the crawler behavior detection process so that it can be reused for the next crawler behavior detection task. In this way, this application can continuously optimize and improve the detection accuracy of the BiLSTM+Attention network for crawler behavior, which can solve the problems of low accuracy and easy misjudgment of crawler behavior detection in existing networks.
[0065] The following embodiments will focus on describing the steps in this application for obtaining the corresponding behavioral feature vector from the raw data. See [link to documentation]. Figure 2 As shown in the figure, an embodiment of the present invention discloses a method for detecting crawler behavior, including:
[0066] Step S21: Preprocess the original data to obtain preprocessed data.
[0067] In this embodiment, the original data can be cleaned, transformed, and preprocessed to obtain the preprocessed data.
[0068] Step S22: Extract the feature information of the preprocessed data to obtain standard data.
[0069] In this embodiment, after obtaining the preprocessed data, key feature information can be extracted. It is understood that, in a specific embodiment, data operation behaviors in the access data can be categorized into operations such as add, delete, modify, query, import, and export, while other operations such as login and viewing data can be categorized into GET and POST. This yields standard data corresponding to user behavior. Furthermore, the standard data may include: username, user IP address, location corresponding to the IP address, user behavior, and behavior time.
[0070] Step S23: Vectorize the user behavior and behavior time in the standard data based on one-hot encoding to obtain the corresponding behavior vector and time weight vector.
[0071] In this embodiment, after obtaining the standard data, the user behavior and behavior time in the standard data can be vectorized based on one-hot encoding. It can be understood that, in a specific embodiment, the user behavior data in the standard data can be vectorized using an 8-dimensional vector. From the 0th to the 7th bit, the operation behaviors listed in step 22 can be represented respectively, thus obtaining the behavior vector.
[0072] Accordingly, in one specific embodiment, the behavioral time data in the standard data can be vectorized using a 26-dimensional vector. Specifically, the time weight can be calculated by the interval between the last access time and the current access time for the same user and IP address. The calculation formula is as follows:
[0073]
[0074] Where T is the time interval weight, and t is the actual time interval between two accesses. If the time interval is greater than 25 seconds, it is considered that the previous access has no impact on the current access, that is, the value is 0; thus, the time weight vector can be obtained.
[0075] Step S24: Concatenate and fuse the behavior vector and the time weight vector to obtain the access behavior vector, and concatenate the access behavior vector and the corresponding preset number of user behavior information before and after the user behavior to obtain the behavior feature vector.
[0076] In this embodiment, after obtaining the behavior vector and the time weight vector, the behavior vector and the time weight vector can be concatenated and fused to obtain the access behavior vector. It is understood that the user behavior corresponding to the access behavior vector is necessarily related to several preceding and following user behaviors, and the impact of preceding and following user behaviors on the current user behavior is different. If the number of included user behaviors is large, it will contain more information, making it difficult to extract behavioral features later; conversely, a small number will lead to overfitting. This application can choose an appropriate number, i.e., an appropriate context window size, to concatenate the access behavior vector to obtain the final behavior feature vector. In a specific embodiment, this application can choose four preceding and four following user behavior information vectors adjacent to the current user behavior to concatenate the access behavior vector corresponding to the current user behavior. The resulting behavior feature vector can include preceding and following access behavior feature information. Figure 3 As shown, the behavior feature vectors that are adjacent to the current user behavior can be concatenated with the access behavior vector corresponding to the current user behavior to obtain a behavior feature vector that includes context information.
[0077] Step S25: Normalize the similarity matrix corresponding to the behavior feature vector to obtain the corresponding normalized similarity matrix, and perform sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior screening results corresponding to the original data.
[0078] Step S26: Use a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results.
[0079] The specific processes of steps S21, S25, and S26 can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.
[0080] Therefore, the embodiments of this application can preprocess the original user access behavior data to obtain standard data, and then vectorize the user behavior and access time interval based on one-hot encoding. This can vectorize discrete and irregular information and then fuse it into a behavior vector for each access by concatenation. Finally, by fusing the context behavior information vector of a single user access, the final behavior feature vector is formed. This allows for the extraction of deeper and more comprehensive access behavior information.
[0081] See Figure 3 As shown in the figure, an embodiment of the present invention discloses a method for detecting crawler behavior, including:
[0082] Step S31: Obtain raw data of user access behavior.
[0083] Step S32: Process the original data according to the preset data processing rules to obtain standard data, and perform vectorization processing on the standard data according to the preset vectorization rules to obtain the corresponding behavioral feature vector.
[0084] Step S33: Calculate the similarity matrix between the behavioral feature vectors, and normalize the similarity matrix to obtain the normalized similarity matrix.
[0085] In this embodiment, after obtaining the behavioral feature vectors, the similarity matrix W between the behavioral feature vectors can be calculated as follows:
[0086]
[0087] Where w(x) i x j ) represents the element in row i and column j of matrix W, x i Let x represent the vector of the i-th user behavior. j Let X represent the vector of the j-th user behavior. ik and X jk Let represent the k-th position in the i-th user behavior vector and the j-th user behavior vector, respectively, and n represent the vector dimension.
[0088] In this embodiment, the normalization process of the similarity matrix corresponding to the behavioral feature vector to obtain the corresponding normalized similarity matrix may specifically include normalizing the similarity matrix, and the corresponding calculation formula is as follows:
[0089]
[0090] in, Let D be the normalized similarity matrix, and D be the degree matrix of the similarity matrix, which is a diagonal matrix. Further, the formula for calculating the diagonal elements is as follows:
[0091]
[0092] Step S34: Perform sparse non-negative matrix decomposition on the normalized similarity matrix according to the preset matrix decomposition function to obtain the initial crawler behavior screening results.
[0093] In this embodiment, after obtaining the normalized similarity matrix, the normalized similarity matrix can be sparsely decomposed using the preset matrix factorization function to obtain the initial crawler behavior screening results. It should be noted that the preset matrix factorization function includes preset risk levels and preset clustering indicator matrices. Specifically, the number n of crawler behavior risk levels can be preset. pSimultaneously, a random initialization clustering indicator matrix H is set. In a specific embodiment, the number of risk levels can be two, i.e., high risk and low risk levels. Then, the preset matrix factorization function can be obtained, and the corresponding function formula is as follows:
[0094]
[0095] Here, the rows of H represent user behavior vectors. It can be understood that by minimizing the aforementioned preset matrix factorization function, a low-rank approximation matrix H of the similarity matrix can be calculated, where the column corresponding to the maximum value in each row of matrix H represents the corresponding crawler behavior risk level. It can be understood that using the preset matrix factorization function to perform sparse non-negative matrix factorization on the similarity matrix can obtain the initial crawler behavior filtering results corresponding to the original data. It can be understood that sparse non-negative matrix factorization can perform unsupervised partitioning of large amounts of unlabeled data, and can handle the problem of high access traffic.
[0096] Step S35: Use a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results.
[0097] In this embodiment, the preset BiLSTM+Attention network can be used to predict the initial crawler behavior screening results. It is understood that the structure of the preset BiLSTM+Attention network may include a five-layer network, namely an input layer, an embedding layer, a BiLSTM layer, an Attention layer, and an output layer. Furthermore, when using the preset BiLSTM+Attention network for prediction, softmax (normalized exponential function) can be used in the output layer to predict the score, and the category with the highest score is taken as the output category, which is the crawler behavior prediction result.
[0098] In one specific embodiment, such as Figure 5 As shown, we can perform a layer of detection and filtering on the behavioral feature vectors corresponding to the original data of user access behavior based on sparse nonnegative matrix factorization, i.e., SNMF, to obtain the initial filtering result, i.e., the set of users with suspected crawling behavior in the figure; then, we can perform secondary processing and prediction through a pre-built BiLSTM+Attention network, so as to obtain the accurate prediction result, i.e., the final crawling users in the figure.
[0099] Furthermore, in another specific embodiment, such as Figure 6As shown, standard data is obtained by preprocessing the original data, and then the standard data is vectorized based on one-hot encoding to obtain a behavior feature vector containing user behavior context information. Preliminary crawler behavior detection results can be obtained based on sparse non-negative matrix factorization. Then, the final prediction result can be obtained by secondary processing and prediction through the constructed preset BiLSTM+Attention network, which is the final user with crawler behavior in the figure.
[0100] The specific processes of steps S31 and S32 described above can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.
[0101] Therefore, the embodiments of this application can use a preset matrix factorization function to perform sparse non-negative matrix factorization on the normalized similarity matrix obtained from the behavioral feature vector. This can perform unsupervised partitioning on a large amount of unlabeled data and obtain the partitioning result, which is the initial screening and detection result. Then, the preset BiLSTM+Attention network can be used to perform secondary prediction on the obtained initial screening and detection result, which can improve the accuracy of the result.
[0102] like Figure 7 As shown, this application provides a web crawler behavior detection device, including:
[0103] Data acquisition module 11 is used to acquire raw data of user access behavior;
[0104] The data vectorization module 12 is used to process the original data according to preset data processing rules to obtain standard data, and to convert the standard data into corresponding behavioral feature vectors based on preset vectorization rules.
[0105] The filtering result determination module 13 is used to normalize the similarity matrix corresponding to the behavior feature vector to obtain the corresponding normalized similarity matrix, and to perform sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior filtering result corresponding to the original data.
[0106] The prediction result determination module 14 is used to predict the initial crawler behavior screening results using a preset BiLSTM+Attention network to obtain the crawler behavior prediction result.
[0107] Therefore, this application can vectorize the original data, obtain initial screening results based on sparse nonnegative matrix factorization, and then use a pre-defined BiLSTM+Attention network to predict the initial screening results to obtain the final prediction results. In this way, this application can obtain screening results even when dealing with high traffic and unlabeled data, and then perform secondary detection using the pre-defined BiLSTM+Attention network, which can improve the accuracy of crawler behavior detection.
[0108] In one specific embodiment, the data vectorization module 12 may include:
[0109] A data preprocessing unit is used to preprocess the raw data to obtain preprocessed data;
[0110] A standard data determination unit is used to extract feature information from the preprocessed data to obtain the standard data; wherein the standard data includes: username, user IP address, IP address location, user behavior, and behavior time.
[0111] In another specific embodiment, the data vectorization module 12 may include:
[0112] An initial vector determination unit is used to vectorize the user behavior and the behavior time in the standard data based on one-hot encoding to obtain the corresponding behavior vector and time weight vector.
[0113] The feature vector determination unit is used to concatenate and fuse the behavior vector and the time weight vector to obtain the access behavior vector, and to concatenate the access behavior vector and a preset number of user behavior information before and after the corresponding user behavior to obtain the behavior feature vector.
[0114] In one specific embodiment, the screening result determination module 13 may include:
[0115] A normalized matrix determination unit is used to calculate the similarity matrix between the behavioral feature vectors and normalize the similarity matrix to obtain the normalized similarity matrix.
[0116] The screening result determination unit is used to perform sparse non-negative matrix decomposition on the normalized similarity matrix according to a preset matrix decomposition function to obtain the initial crawler behavior screening result; wherein, the preset matrix decomposition function includes a preset risk level and a preset clustering indicator matrix.
[0117] In one specific embodiment, the prediction result determination module 14 may include:
[0118] The initial result determination unit is used to predict the initial crawler behavior screening result in the preset BiLSTM+Attention network using a normalized exponential function to obtain the initial prediction result.
[0119] The final result determination unit is used to determine the category with the highest score in the initial prediction results as the crawler behavior prediction result.
[0120] In another specific embodiment, the prediction result determination module 14 may further include:
[0121] The network training unit is used to train the BiLSTM+Attention network using the behavioral feature vector to obtain the preset BiLSTM+Attention network.
[0122] In yet another specific embodiment, the prediction result determination module 14 may further include:
[0123] The sample determination unit is used to determine the crawler behavior data and non-crawler behavior data in the initial crawler behavior screening results as positive samples and negative samples, respectively.
[0124] The network optimization unit is used to train and optimize the preset BiLSTM+Attention network using the positive samples and the negative samples, and to determine the optimized BiLSTM+Attention network as the preset BiLSTM+Attention network for the next use.
[0125] Furthermore, in one specific embodiment, the prediction result determination module 14 may further include:
[0126] The test data determination unit is used to determine the crawler behavior data in the initial crawler behavior screening results as test data.
[0127] The network evaluation unit is used to verify the optimized BiLSTM+Attention network using the test data to obtain the corresponding evaluation results.
[0128] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0129] Figure 8This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the crawler behavior detection method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.
[0130] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0131] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0132] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the crawler behavior detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0133] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned web crawler behavior detection method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0137] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0138] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for detecting crawler behavior, characterized in that, include: Obtain raw data on user access behavior; The original data is processed according to preset data processing rules to obtain standard data, and the standard data is vectorized according to preset vectorization rules to obtain corresponding behavioral feature vectors. The similarity matrix corresponding to the behavior feature vector is normalized to obtain the corresponding normalized similarity matrix, and the normalized similarity matrix is subjected to sparse non-negative matrix decomposition to obtain the initial crawler behavior screening results corresponding to the original data. The initial crawler behavior screening results are predicted using a preset BiLSTM+Attention network to obtain the crawler behavior prediction results. The step of processing the raw data according to preset data processing rules to obtain standard data includes: The original data is preprocessed to obtain preprocessed data; The feature information of the preprocessed data is extracted to obtain the standard data; wherein the standard data includes: username, user IP address, location corresponding to the IP address, user behavior, and behavior time. The step of vectorizing the standard data based on preset vectorization rules to obtain the corresponding behavioral feature vectors includes: The user behavior and the behavior time in the standard data are vectorized based on one-hot encoding to obtain the corresponding behavior vector and time weight vector. The behavior vector and the time weight vector are concatenated and fused to obtain the access behavior vector, and the access behavior vector and a preset number of user behavior information before and after the corresponding user behavior are concatenated to obtain the behavior feature vector. The process of normalizing the similarity matrix corresponding to the behavior feature vector to obtain a corresponding normalized similarity matrix, and then performing sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior screening results corresponding to the original data, includes: Calculate the similarity matrix between the behavioral feature vectors, and normalize the similarity matrix to obtain the normalized similarity matrix; The initial crawler behavior screening results are obtained by performing sparse non-negative matrix decomposition on the normalized similarity matrix according to a preset matrix decomposition function; wherein, the preset matrix decomposition function includes a preset risk level and a preset clustering indicator matrix.
2. The crawler behavior detection method according to claim 1, characterized in that, The step of using a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain crawler behavior prediction results includes: The initial prediction result is obtained by using the normalized exponential function to predict the initial crawler behavior screening result in the preset BiLSTM+Attention network. The category with the highest score in the initial prediction results is determined as the crawler behavior prediction result.
3. The crawler behavior detection method according to claim 1 or 2, characterized in that, Before using a preset BiLSTM+Attention network to predict the initial crawler behavior screening results to obtain the crawler behavior prediction results, the method further includes: The BiLSTM+Attention network is trained using the behavioral feature vector to obtain the preset BiLSTM+Attention network.
4. The crawler behavior detection method according to claim 3, characterized in that, After obtaining the initial crawler behavior filtering results corresponding to the original data, the process further includes: The crawler behavior data and non-crawler behavior data in the initial crawler behavior screening results are respectively identified as positive samples and negative samples; The preset BiLSTM+Attention network is trained and optimized using the positive and negative samples, and the optimized BiLSTM+Attention network is determined as the preset BiLSTM+Attention network for the next use.
5. The crawler behavior detection method according to claim 4, characterized in that, After obtaining the crawler behavior prediction result by predicting the initial crawler behavior screening result using a preset BiLSTM+Attention network, the method further includes: The crawler behavior data in the initial crawler behavior screening results is determined as test data; The optimized BiLSTM+Attention network is validated using the test data to obtain the corresponding evaluation results.
6. A crawler behavior detection device, characterized in that, include: The data acquisition module is used to acquire raw data on user access behavior; The data vectorization module is used to process the original data according to preset data processing rules to obtain standard data, and to convert the standard data into corresponding behavioral feature vectors based on preset vectorization rules. The filtering result determination module is used to normalize the similarity matrix corresponding to the behavior feature vector to obtain the corresponding normalized similarity matrix, and to perform sparse non-negative matrix decomposition on the normalized similarity matrix to obtain the initial crawler behavior filtering result corresponding to the original data. The prediction result determination module is used to predict the crawler behavior prediction result by using a preset BiLSTM+Attention network to predict the initial crawler behavior screening result. The data vectorization module includes: A data preprocessing unit is used to preprocess the raw data to obtain preprocessed data; A standard data determination unit is used to extract feature information from the preprocessed data to obtain the standard data; wherein the standard data includes: username, user IP address, location corresponding to the IP address, user behavior, and behavior time. The data vectorization module includes: An initial vector determination unit is used to vectorize the user behavior and the behavior time in the standard data based on one-hot encoding to obtain the corresponding behavior vector and time weight vector. The feature vector determination unit is used to concatenate and fuse the behavior vector and the time weight vector to obtain the access behavior vector, and to concatenate the access behavior vector and a preset number of user behavior information before and after the user behavior to obtain the behavior feature vector. The filtering result determination module includes: A normalized matrix determination unit is used to calculate the similarity matrix between the behavioral feature vectors and normalize the similarity matrix to obtain the normalized similarity matrix. The screening result determination unit is used to perform sparse non-negative matrix decomposition on the normalized similarity matrix according to a preset matrix decomposition function to obtain the initial crawler behavior screening result; wherein, the preset matrix decomposition function includes a preset risk level and a preset clustering indicator matrix.
7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the crawler behavior detection method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the crawler behavior detection method as described in any one of claims 1 to 5.