A crawler behavior detection method, device, equipment and storage medium

Abstract

The application discloses a crawler behavior detection method and device, equipment and a storage medium, and relates to the technical field of data leakage prevention. The method comprises the following steps: obtaining original traffic data, and preprocessing the original traffic data to obtain key information tuples; based on the key information tuples, a Count Sketch algorithm and a principal component analysis method are used to determine a dense feature vector of each IP address in the original traffic data; a first mixed integer linear programming model is constructed based on a space dimension, and the first mixed integer linear programming model is used to determine a plurality of suspected crawler IPs corresponding to the dense feature vector, so as to obtain a suspected crawler IP set; and a hidden Markov model is constructed based on a time dimension for the suspected crawler IP set, so as to determine a crawler IP in the suspected crawler IP set group by using the hidden Markov model. Through the technical scheme of the application, the accuracy and real-time performance of crawler detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of data leakage prevention technology, and in particular to a method, apparatus, equipment and storage medium for detecting web crawler behavior. Background Technology

[0002] Data loss prevention is a challenge every enterprise faces in its operations and management. The leakage of critical information can cause enormous losses, damaging the company's image, affecting its stable operation, harming customers, and potentially leading to security incidents such as fraud, extortion, and privacy breaches. Based on the pathways and sources of data leakage, data loss prevention can be divided into: network-side prevention, endpoint-side prevention, and application-side prevention. Currently, enterprises are gradually building data loss prevention capabilities. However, on the application side, strong data loss prevention capabilities are still lacking. Application-side data leakage methods include: credential stuffing attacks, data scraping, and account brute-force attacks. Statistics show that data leakage incidents caused by web scraping account for over 90%, demonstrating that web scraping plays a dominant role in application-side data leakage.

[0003] Traditional anti-scraping techniques include User-Agent (UA) request control, Session access restrictions, spider traps, CAPTCHA verification, dynamic data loading, IP (Internet Protocol) restrictions, and parameter encryption. However, traditional techniques have many problems. Scraping programs can bypass these barriers to obtain resources by forging legitimate program header information or by setting up multiple sessions to poll the target website. This also increases the website's resource consumption and significantly reduces the user experience for legitimate users. Traditional methods also suffer from low real-time performance, weak implementation, and inability to handle large-scale data. Furthermore, most current traditional methods only involve spatial or temporal judgments of the accessed object, resulting in low detection accuracy and a high false positive rate.

[0004] In summary, improving the accuracy and real-time performance of web crawler detection is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for detecting web crawler behavior, which can improve the accuracy and real-time performance of web crawler detection. The specific solution is as follows:

[0006] Firstly, this application discloses a method for detecting web crawler behavior, including:

[0007] Obtain raw traffic data and preprocess the raw traffic data to obtain key information tuples;

[0008] Based on the key information tuple, the dense feature vector of each IP address in the original traffic data is determined using the Count Sketch algorithm and principal component analysis.

[0009] A first mixed-integer linear programming model is constructed based on the spatial dimension, and several suspected crawler IPs corresponding to the dense feature vector are determined using the first mixed-integer linear programming model to obtain a set of suspected crawler IPs.

[0010] A hidden Markov model is constructed based on the time dimension for the suspected crawler IP set, so as to use the hidden Markov model to determine the crawler IPs in the suspected crawler IP set group.

[0011] Optionally, the step of acquiring raw traffic data and preprocessing the raw traffic data to obtain key information tuples includes:

[0012] Obtain raw traffic data, and perform data cleaning and deduplication on the raw traffic data to obtain intermediate processing data;

[0013] Extract key information from the intermediate processing data and store the key information as a tuple to obtain a key information tuple.

[0014] Optionally, the step of determining the dense feature vector of each IP address in the original traffic data based on the key information tuple using the Count Sketch algorithm and principal component analysis includes:

[0015] A feature extraction model is constructed based on the Count Sketch algorithm, and the key information tuples are input into the feature extraction model to obtain daily statistical results;

[0016] For each IP address in the original traffic data, obtain a corresponding consecutive first preset number of daily statistical results, and use the consecutive first preset number of daily statistical results to determine the mean, variance, minimum and maximum values ​​of each IP address;

[0017] The mean, variance, minimum, and maximum values ​​are concatenated to perform feature fusion, resulting in a sparse feature vector for each IP address.

[0018] A feature dimensionality reduction model is constructed based on principal component analysis, and the sparse feature vector is input into the feature dimensionality reduction model to obtain the dense feature vector of each IP address through information aggregation.

[0019] Optionally, the step of constructing a first mixed-integer linear programming model based on the spatial dimension, and using the first mixed-integer linear programming model to determine several suspected crawler IPs corresponding to the dense feature vector, to obtain a set of suspected crawler IPs, includes:

[0020] A decision variable set is constructed based on the dense feature vector; the decision variable set includes a first decision variable for determining whether the IP address in the original traffic data belongs to the suspected crawler IP, a second decision variable for determining whether the IP address belongs to at least one suspected crawler IP in the suspected crawler IP set, a third decision variable for determining whether the IP address belongs to multiple suspected crawler IPs in the suspected crawler IP set, and a fourth decision variable for representing the first second preset number of IP addresses in the suspected crawler IP set;

[0021] A target function is constructed based on the dense feature vectors; the target function is: N p t is the number of the pre-set set of suspected crawler IPs. i Let N be the fourth decision variable. i For the second preset quantity, var(p) i ) is a set p i The first N i The variance of each IP address;

[0022] A set of constraints is constructed based on the dense feature vectors; the set of constraints includes constraints for constraining the similarity and adjacency relationships between the suspected crawler IPs;

[0023] By using a preset solver software to solve the decision variable set, the objective function, and the set of constraints, several suspected crawler IPs corresponding to the dense feature vector are determined to obtain a set of suspected crawler IPs.

[0024] Optionally, after constructing a first mixed-integer linear programming model based on the spatial dimension and using the first mixed-integer linear programming model to determine several suspected crawler IPs in the dense feature vector to obtain a set of suspected crawler IPs, the method further includes:

[0025] The mixed-integer linear programming model is reconstructed based on the spatial dimension to obtain a second mixed-integer linear programming model;

[0026] The IP addresses corresponding to the dense feature vectors are reassigned using the second mixed-integer linear programming model to obtain an optimized set of target suspected crawler IPs.

[0027] Optionally, the step of constructing a Hidden Markov Model based on the time dimension for the suspected crawler IP set, and using the Hidden Markov Model to determine the crawler IPs in the suspected crawler IP set, includes:

[0028] The three essential elements for constructing the Hidden Markov Model are determined using a first preset algorithm.

[0029] The second preset algorithm determines the crawler IPs in the suspected crawler IP set group based on the three elements of the model.

[0030] Optionally, the step of determining the three model elements used to construct the Hidden Markov Model through a first preset algorithm includes:

[0031] The three essential elements for constructing the Hidden Markov Model are determined using the Baum-Welch algorithm.

[0032] Furthermore, the step of determining the crawler IPs in the suspected crawler IP set group based on the three elements of the model using the second preset algorithm includes:

[0033] The Viterbi algorithm is used to determine the crawler IPs in the suspected crawler IP set based on the three elements of the model.

[0034] Secondly, this application discloses a web crawler behavior detection device, comprising:

[0035] The raw traffic data acquisition module is used to acquire raw traffic data and preprocess the raw traffic data to obtain key information tuples.

[0036] The feature extraction module is used to determine the dense feature vector of each IP address in the original traffic data based on the key information tuple, using the Count Sketch algorithm and principal component analysis.

[0037] The spatial dimension modeling module is used to construct a first mixed-integer linear programming model based on the spatial dimension, and to use the first mixed-integer linear programming model to determine a number of suspected crawler IPs corresponding to the dense feature vector, so as to obtain a set of suspected crawler IPs.

[0038] The time dimension module is used to construct a hidden Markov model based on the time dimension for the suspected crawler IP set, so as to use the hidden Markov model to determine the crawler IPs in the suspected crawler IP set group.

[0039] Thirdly, this application discloses an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the crawler behavior detection method as described above.

[0040] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the crawler behavior detection method as described above.

[0041] This application discloses a method for detecting web crawler behavior. First, raw traffic data is acquired and preprocessed to obtain key information tuples. Then, based on these key information tuples, the Count Sketch algorithm and Principal Component Analysis (PCA) are used to determine the dense feature vector of each IP address in the raw traffic data. Next, a first mixed-integer linear programming model is constructed based on the spatial dimension, and this model is used to determine several suspected web crawler IPs corresponding to the dense feature vectors, thus obtaining a set of suspected web crawler IPs. Finally, a Hidden Markov Model (HMM) is constructed based on the time dimension for the set of suspected web crawler IPs, and this HMM is used to determine the web crawler IPs within the suspected web crawler IP set. It is evident that the Count Sketch algorithm provides strong support for big data processing, and the feature extraction method based on the Count Sketch algorithm and PCA can provide dense feature vectors for each IP's access traffic, thereby extracting deeper and more comprehensive access information. Furthermore, this invention integrates multi-dimensional information modeling of IP access using a stacking method, performing two analyses and judgments: First, spatial dimension modeling is performed using a mixed-integer linear programming model to filter out the set of suspected crawler IPs from dense feature vectors, solving the problem of high access traffic and lack of data labels; then, temporal dimension modeling is performed using a hidden Markov model to identify the crawler IPs in the suspected crawler IP set, solving the problem of low accuracy and frequent misjudgments in traditional crawler detection methods. Compared to using a single basic model for crawler IP detection, this invention has higher detection accuracy and real-time performance.

[0042] In addition, the crawler behavior detection device, equipment and storage medium provided in this application correspond to the above-mentioned crawler behavior detection method and have the same effect. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a web crawler behavior detection method disclosed in this application;

[0045] Figure 2 This is a schematic diagram of a feature extraction structure based on Count Sketch and PCA disclosed in this application;

[0046] Figure 3 This is a schematic diagram of a crawler behavior detection method based on multi-dimensional access information disclosed in this application;

[0047] Figure 4 This is a flowchart of a specific web crawler behavior detection method disclosed in this application;

[0048] Figure 5 This is a schematic diagram of a spatiotemporal access pattern modeling based on MILP and HMM disclosed in this application;

[0049] Figure 6 This is a schematic diagram of the structure of a crawler behavior detection device disclosed in this application;

[0050] Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Currently, traditional anti-scraping technologies have many problems. During crawler detection, crawlers can bypass barriers and obtain resources by forging legitimate program header information or polling the target website using multiple sessions. This increases website resource consumption and significantly reduces the user experience for legitimate users. Traditional methods also suffer from low real-time performance, weak implementation, and inability to handle large-scale data. Furthermore, most traditional methods only involve spatial or temporal judgments of the accessed object, resulting in low detection accuracy and a high false positive rate.

[0053] Therefore, this application provides a web crawler behavior detection scheme that can improve the accuracy and real-time performance of web crawler detection.

[0054] This invention discloses a method for detecting crawler behavior, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0055] Step S11: Obtain raw traffic data and preprocess the raw traffic data to obtain key information tuples.

[0056] In this embodiment, raw IP access traffic data is prepared, preprocessed, and a standard data structure is constructed. Specifically, raw traffic data is acquired, and the raw traffic data is cleaned and deduplicated to obtain intermediate processing data; key information is extracted from the intermediate processing data, and the key information is stored in tuples to obtain key information tuples.

[0057] It is understandable that after obtaining the raw traffic data, it is cleaned and deduplicated. The raw traffic data after cleaning and deduplication is used as intermediate processing data and key information is extracted from it. The extracted key information includes, but is not limited to, source address, destination address, source port, destination port, protocol, traffic, time, etc. The extracted key information is represented by N-tuples as a standard data structure, that is, key information tuples.

[0058] Step S12: Based on the key information tuple, use the Count Sketch algorithm and principal component analysis to determine the dense feature vector of each IP address in the original traffic data.

[0059] Because traditional methods suffer from low real-time performance and limited applicability in web crawler detection, and are unable to handle large-scale data, this application addresses these shortcomings by first utilizing the Count Sketch algorithm for data cleaning and feature extraction. This provides strong support for big data processing through the Count Sketch algorithm. Specifically, a feature extraction model is constructed based on the Count Sketch algorithm, and the key information tuples are input into the feature extraction model to obtain daily statistical results.

[0060] In this embodiment, the key information tuple obtained in step S11 is input into a feature extraction model built based on the CountSketch algorithm to obtain the daily statistical result Sketch. It should be noted that the daily statistical result Sketch is a two-dimensional array with w columns and d rows, where the parameters w and d are hyperparameters defined in advance based on experience and are related to the final query error rate. Alternatively, global statistical results can also be obtained through the feature extraction model; this embodiment uses the daily statistical results.

[0061] Furthermore, for each IP address in the original traffic data, a corresponding first preset number of daily statistical results are obtained, and the mean, variance, minimum, and maximum values ​​of each IP address are determined using the first preset number of daily statistical results. The mean, variance, minimum, and maximum values ​​are then concatenated to perform feature fusion, resulting in a sparse feature vector for each IP address. A feature dimensionality reduction model is constructed based on principal component analysis, and the sparse feature vector is input into the feature dimensionality reduction model to obtain a dense feature vector for each IP address through information aggregation.

[0062] In this embodiment, for each IP address, a first preset number of consecutive daily statistical results are obtained from the acquired daily statistical results (Sketch) information, such as obtaining five consecutive days' Sketches, i.e., five two-dimensional arrays. Then, the mean, variance, minimum, and maximum values ​​are calculated point-to-point to generate four new Sketches. These four new Sketches (mean, variance, minimum, and maximum values) are then concatenated to perform feature fusion, obtaining a sparse feature vector for each IP address. Finally, a feature dimensionality reduction model based on Principal Component Analysis (PCA) is constructed, inputting the obtained sparse feature vector for each IP address, and obtaining a denser feature vector for each IP address through information aggregation.

[0063] like Figure 2 The diagram shows the structure of feature extraction based on Count Sketch and PCA. The original data is pre-processed to obtain a fixed structure. Then, sparse feature vectors are obtained by Count Sketch feature extraction. Finally, dense feature vectors for each IP access traffic are obtained by PCA feature dimensionality reduction. Experiments can verify that this method can extract deeper and more comprehensive access information.

[0064] Step S13: Construct a first mixed-integer linear programming model based on the spatial dimension, and use the first mixed-integer linear programming model to determine a number of suspected crawler IPs corresponding to the dense feature vector, so as to obtain a set of suspected crawler IPs.

[0065] In this embodiment of the application, the accuracy and false positive rate of judging access objects using a single model that uses spatial or temporal dimensions are not high. Therefore, the two-level detection model is fused based on a stacking and integration strategy to obtain the final crawler behavior detection model, which can have higher detection accuracy and real-time performance than using a single model.

[0066] In this embodiment, a spatial dimension model of all accessed IPs is first performed using a Mixed Integer Linear Programming (MILP) model to obtain a set of suspected crawler IPs. That is, a first-level crawler IP filter based on MILP is constructed, and a set of suspected crawler IPs is obtained by modeling the spatial dimension of IP access. The number of suspected crawler IPs can be multiple, and is generally pre-set, with a default of three: a highly suspected crawler IP set, a moderately suspected crawler IP set, and a slightly suspected crawler IP set.

[0067] It is understandable that in the process of determining the suspected crawler IP set through MILP modeling, due to the distance settings, some virtual IPs of suspected crawlers may not be included in the set, or some IP addresses may be assigned to multiple suspected crawler IP sets. Therefore, in this embodiment, the MILP concept is used for secondary modeling and optimization to redistribute IP addresses and obtain the final filtering result, thus obtaining the optimized target suspected crawler IP set.

[0068] Step S14: Construct a Hidden Markov Model based on the time dimension for the suspected crawler IP set, and use the Hidden Markov Model to determine the crawler IPs in the suspected crawler IP set group.

[0069] In this embodiment, after spatial dimensional modeling of all accessed IPs using MILP to obtain a set of suspected crawler IPs, a Hidden Markov Model (HMM) is used to model the temporal dimension of the crawler IPs in the set to obtain the detection results. Furthermore, this application verifies through ablation learning and comparative experiments that the model has higher detection accuracy and real-time performance than using a single model.

[0070] like Figure 3 The diagram illustrates the crawler behavior detection based on multi-dimensional access information in this invention. First, raw IP access traffic data is preprocessed to construct a standard data structure. Second, a feature extraction module based on Count Sketch is built, using PCA feature dimensionality reduction to obtain traffic representation vectors and dense feature vectors of IP access. Then, a first-level crawler IP filter based on MILP is constructed, obtaining a set of suspected crawler IPs by modeling the spatial dimension of IP access. Finally, a second-level crawler IP determiner based on HMM is constructed, identifying crawler IPs in the set by modeling the temporal dimension of IP access. In this way, a stacked fusion strategy integrates the two-level models to obtain the final crawler behavior detection model that identifies the crawler IPs.

[0071] This application discloses a method for detecting web crawler behavior. First, raw traffic data is acquired and preprocessed to obtain key information tuples. Then, based on these key information tuples, the Count Sketch algorithm and Principal Component Analysis (PCA) are used to determine the dense feature vector of each IP address in the raw traffic data. Next, a first mixed-integer linear programming model is constructed based on the spatial dimension, and this model is used to determine several suspected web crawler IPs corresponding to the dense feature vectors, thus obtaining a set of suspected web crawler IPs. Finally, a Hidden Markov Model (HMM) is constructed based on the time dimension for the set of suspected web crawler IPs, and this HMM is used to determine the web crawler IPs within the suspected web crawler IP set. It is evident that the Count Sketch algorithm provides strong support for big data processing, and the feature extraction method based on the Count Sketch algorithm and PCA can provide dense feature vectors for each IP's access traffic, thereby extracting deeper and more comprehensive access information. Furthermore, this invention integrates multi-dimensional information modeling of IP access using a stacking method, performing two analyses and judgments: First, spatial dimension modeling is performed using a mixed-integer linear programming model to filter out the set of suspected crawler IPs from dense feature vectors, solving the problem of high access traffic and lack of data labels; then, temporal dimension modeling is performed using a hidden Markov model to identify the crawler IPs in the suspected crawler IP set, solving the problem of low accuracy and frequent misjudgments in traditional crawler detection methods. Compared to using a single basic model for crawler IP detection, this invention has higher detection accuracy and real-time performance.

[0072] This application discloses a specific method for detecting crawler behavior. See also: Figure 4 As shown, the method includes:

[0073] Step S21: Obtain raw traffic data and preprocess the raw traffic data to obtain key information tuples.

[0074] Step S22: Based on the key information tuple, use the Count Sketch algorithm and principal component analysis to determine the dense feature vector of each IP address in the original traffic data.

[0075] For more detailed processing procedures regarding steps S21 and S22, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0076] Step S23: Construct a decision variable set, an objective function, and a set of constraints based on the dense feature vector; solve the decision variable set, the objective function, and the set of constraints using a preset solver software to determine several suspected crawler IPs corresponding to the dense feature vector, so as to obtain a set of suspected crawler IPs.

[0077] In this embodiment, secondary modeling is performed based on the MILP concept. MILP modeling includes decision variables, objective function, and constraints. In the initial MILP modeling, a set of decision variables, an objective function, and a set of constraints are constructed based on the dense feature vectors.

[0078] In this embodiment of the application, there are four decision variables in the decision variable group, including (1) a first decision variable used to determine whether the IP address in the original traffic data belongs to the suspected crawler IP; such as x ij Indicates whether the j-th IP address belongs to the i-th suspected crawler IP in the suspected crawler IP set. (2) A second decision variable used to determine whether the IP address belongs to at least one suspected crawler IP in the suspected crawler IP set; such as x′ i Indicates whether the j-th IP address belongs to at least one of the suspected crawler IPs in the suspected crawler IP set. (3) A third decision variable used to determine whether the IP address belongs to multiple suspected crawler IPs in the suspected crawler IP set; such as x″ i Indicates whether the j-th IP address belongs to multiple suspected crawler IPs in the suspected crawler IP set. (4) A fourth decision variable used to characterize the first second preset number of IP addresses in the suspected crawler IP set; such as t i This represents the top N IPs in the set of the i-th suspected crawler IPs. i The weighted variance of each IP address.

[0079] In this embodiment, the number of suspected crawler IPs can be multiple, and is generally pre-set, with a default of three: a highly suspected crawler IP set, a moderately suspected crawler IP set, and a slightly suspected crawler IP set. The objective function is to minimize the weighted sum of the variances of the feature vectors of the selected suspected crawler IP sets, and the specific formula is as follows: Where, N p The number of suspected crawler IPs set as preset; t i Let be the fourth decision variable, that is, represent the top N IPs in the set of the i-th suspected crawler IPs. i The weighted variance of each IP address, t i =N i ×var(P i );N i The second preset quantity; var(p) i) is a set p i The first N i The variance of each IP address.

[0080] In this embodiment, a set of constraints is constructed based on the dense feature vectors. The set of constraints includes constraints for constraining the similarity and adjacency relationships between the suspected crawler IPs. Therefore, the constraints for the initial MILP modeling need to ensure that the selected suspected crawler IPs have high similarity and adjacency relationships, as shown in the following formula:

[0081]

[0082]

[0083] ∑ j∈ζ x′ j ≥a′×N;

[0084]

[0085] ∑ j∈ζ x″ j ≤a″×N;

[0086]

[0087]

[0088]

[0089] x ij ∈{0,1},x′ i ∈{0,1},x″ j ∈{0,1},t i ∈R;

[0090] Where, x ij Indicates whether the j-th IP address belongs to the i-th suspected crawler IP in the set of suspected crawler IPs, R i (j) represents the j-th feature in the feature vector x of the i-th IP address, and N represents the number of IP addresses; x′ j This indicates whether the j-th IP address belongs to at least one of the suspected crawler IPs in the suspected crawler IP set; ζ represents the set of all accessing IP addresses in the network; a′ represents the proportion of IP addresses that are assigned to at least one suspected crawler IP set out of the total number of IP addresses, typically taking a value of 0.4; x″ j This indicates whether the j-th IP address belongs to multiple suspected crawler IPs in the suspected crawler IP set; a″ represents the proportion of IP addresses assigned to multiple suspected crawler IP sets out of the total number of IP addresses, typically taking a value of 0.1; N minThis indicates the minimum number of IP addresses contained in each suspected crawler IP cluster; (j) represents the position index of the j-th feature in the i-th IP address, var(P {i，j} ) represents the variance of the combined feature parameters of the first j features of the i-th IP address; N p This represents the number of suspected crawler IP addresses.

[0091] In the initial modeling stage, the decision variables, objective function, and constraints of MILP are used to optimize the solution using existing solver software, obtaining preliminary filtering results for the suspected crawler IP set. Further, based on the preliminary filtering results, a second modeling and optimization process is performed using the MILP approach. This involves reallocating some IP addresses that were initially assigned to multiple suspected crawler IP sets, thereby obtaining an optimized target suspected crawler IP set and the final filtering result.

[0092] Step S24: Determine the three model elements used to construct the Hidden Markov Model using the first preset algorithm; determine the crawler IPs in the suspected crawler IP set group using the second preset algorithm based on the three model elements.

[0093] In this embodiment, a two-level crawler IP determiner based on a Hidden Markov Model (HMM) is constructed. This is achieved by performing time-dimensional modeling on each IP in the suspected crawler IP set obtained in step S23, thereby ultimately determining whether the IPs in the set are crawler IPs. Specifically, the modeling process based on the HMM first determines the three key elements of the model using the Baum-Welch algorithm: λ = (A, B, π). Here, A represents the transition probability matrix of IPs identified as suspected crawler IPs; B represents the probability distribution matrix generated by directly observable access patterns, where the access patterns represent the probability distribution values ​​of each element in the matrix; and π represents the initial distribution matrix of IPs identified as suspected crawler IPs. Finally, the Viterbi algorithm is used to obtain the final prediction result based on the three key elements of the model, i.e., whether the IP address is a crawler IP.

[0094] like Figure 5 The diagram illustrates spatiotemporal access pattern modeling based on MILP and HMM. Dense feature vectors obtained using Count Sketch and Principal Component Analysis are input into the MILP-based model to obtain a first-level crawler IP filter. During this process, a second-level modeling is performed to obtain a set of suspected crawler IPs. Then, an HMM model is constructed on this set of suspected crawler IPs to build a second-level crawler IP determiner, which then determines the final crawler IPs.

[0095] This application discloses a method for detecting web crawler behavior. First, raw traffic data is acquired and preprocessed to obtain key information tuples. Then, based on these key information tuples, a dense feature vector for each IP address in the raw traffic data is determined using the Count Sketch algorithm and Principal Component Analysis (PCA). Next, a set of decision variables, an objective function, and a set of constraints are constructed based on the dense feature vectors. The decision variable set, objective function, and constraint set are solved using a pre-defined solver software to identify several suspected web crawler IPs corresponding to the dense feature vectors, thus obtaining a set of suspected web crawler IPs. Finally, a first pre-defined algorithm determines the three model elements used to construct the Hidden Markov Model (HMM). A second pre-defined algorithm determines the web crawler IPs in the suspected web crawler IP set based on the three model elements. It is evident that the Count Sketch algorithm provides strong support for big data processing. The feature extraction method based on the Count Sketch algorithm and PCA can provide a dense feature vector for each IP's access traffic, thereby extracting deeper and more comprehensive access information. Furthermore, this invention integrates multi-dimensional information modeling of IP access using a stacking method, performing two analyses and judgments: First, spatial dimension modeling is performed using a mixed-integer linear programming model to filter out the set of suspected crawler IPs from dense feature vectors, solving the problem of high access traffic and lack of data labels; then, temporal dimension modeling is performed using a hidden Markov model to identify the crawler IPs in the suspected crawler IP set, solving the problem of low accuracy and frequent misjudgments in traditional crawler detection methods. Compared to using a single basic model for crawler IP detection, this invention has higher detection accuracy and real-time performance.

[0096] Accordingly, this application also discloses a web crawler behavior detection device, see [link to relevant documentation]. Figure 6 As shown, the device includes:

[0097] The raw traffic data acquisition module 11 is used to acquire raw traffic data and preprocess the raw traffic data to obtain key information tuples.

[0098] The feature extraction module 12 is used to determine the dense feature vector of each IP address in the original traffic data based on the key information tuple using the Count Sketch algorithm and principal component analysis.

[0099] The spatial dimension modeling module 13 is used to construct a first mixed integer linear programming model based on the spatial dimension, and use the first mixed integer linear programming model to determine a number of suspected crawler IPs corresponding to the dense feature vector, so as to obtain a set of suspected crawler IPs.

[0100] The time dimension construction module 14 is used to construct a hidden Markov model based on the time dimension for the suspected crawler IP set, so as to use the hidden Markov model to determine the crawler IPs in the suspected crawler IP set group.

[0101] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0102] Therefore, the above-described scheme in this embodiment first obtains raw traffic data and preprocesses it to obtain key information tuples. Then, based on the key information tuples, the Count Sketch algorithm and Principal Component Analysis (PCA) are used to determine the dense feature vector of each IP address in the raw traffic data. Next, a first mixed-integer linear programming model is constructed based on the spatial dimension, and the first mixed-integer linear programming model is used to determine several suspected crawler IPs corresponding to the dense feature vectors, thus obtaining a set of suspected crawler IPs. Finally, a Hidden Markov Model (HMM) is constructed based on the time dimension for the set of suspected crawler IPs, and the HMM is used to determine the crawler IPs in the suspected crawler IP set. It is evident that the Count Sketch algorithm provides strong support for big data processing. The feature extraction method based on the Count Sketch algorithm and PCA can provide dense feature vectors for each IP access traffic, thereby extracting deeper and more comprehensive access information. Furthermore, this invention integrates multi-dimensional information modeling of IP access using a stacking method, performing two analyses and judgments: First, spatial dimension modeling is performed using a mixed-integer linear programming model to filter out the set of suspected crawler IPs from dense feature vectors, solving the problem of high access traffic and lack of data labels; then, temporal dimension modeling is performed using a hidden Markov model to identify the crawler IPs in the suspected crawler IP set, solving the problem of low accuracy and frequent misjudgments in traditional crawler detection methods. Compared to using a single basic model for crawler IP detection, this invention has higher detection accuracy and real-time performance.

[0103] Furthermore, embodiments of this application also disclose an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0104] Figure 7This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the crawler behavior detection method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0105] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0106] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0107] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the crawler behavior detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0108] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. The computer program, when executed by a processor, implements the aforementioned crawler behavior detection method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0110] The steps of the crawler behavior detection or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0111] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The foregoing has provided a detailed description of the crawler behavior detection method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting crawler behavior, characterized in that, The method comprises the following steps: obtaining original traffic data and preprocessing the original traffic data to obtain key information tuples; the preprocessing is used to construct a standard data structure in the form of N-tuple representation; based on the key information tuples, a dense feature vector of each IP address in the original traffic data is determined by using a Count Sketch algorithm and a principal component analysis method; a first mixed integer linear programming model is constructed based on a spatial dimension, and a plurality of suspected crawler IPs corresponding to the dense feature vector are determined by using the first mixed integer linear programming model to obtain a suspected crawler IP set; a hidden Markov model is constructed based on a time dimension for the suspected crawler IP set to determine crawler IPs in the suspected crawler IP set by using the hidden Markov model; after the first mixed integer linear programming model is constructed based on the spatial dimension and the plurality of suspected crawler IPs corresponding to the dense feature vector are determined by using the first mixed integer linear programming model to obtain the suspected crawler IP set, the method further comprises the following steps: the mixed integer linear programming model is constructed again based on the spatial dimension to obtain a second mixed integer linear programming model; the IP addresses corresponding to the dense feature vector are re-allocated by using the second mixed integer linear programming model to obtain an optimized target suspected crawler IP set.

2. The crawler behavior detection method of claim 1, wherein, The method of obtaining original traffic data and preprocessing the original traffic data to obtain key information tuples comprises the following steps: obtaining original traffic data and performing data cleaning and deduplication on the original traffic data to obtain intermediate processing data; extracting key information from the intermediate processing data and storing the key information in the form of tuples to obtain key information tuples.

3. The crawler behavior detection method of claim 1, wherein, The method of determining a dense feature vector of each IP address in the original traffic data based on the key information tuples by using a Count Sketch algorithm and a principal component analysis method comprises the following steps: a feature extraction model is constructed based on a Count Sketch algorithm, and the key information tuples are input into the feature extraction model to obtain daily statistical results; for each IP address in the original traffic data, a continuous first preset number of daily statistical results are obtained, and the continuous first preset number of daily statistical results are used to determine a mean value result, a variance result, a minimum value result and a maximum value result of each IP address; the mean value result, the variance result, the minimum value result and the maximum value result are fused by splicing to obtain a sparse feature vector of each IP address; a feature dimension reduction model is constructed based on a principal component analysis method, and the sparse feature vector is input into the feature dimension reduction model to obtain a dense feature vector of each IP address by information aggregation.

4. The crawler behavior detection method of claim 1, wherein, The method of constructing a first mixed integer linear programming model based on a spatial dimension and determining a plurality of suspected crawler IPs corresponding to the dense feature vector by using the first mixed integer linear programming model to obtain a suspected crawler IP set comprises the following steps: constructing a decision variable group based on the dense feature vector; the decision variable group includes a first decision variable for judging whether an IP address in the original traffic data belongs to the suspected crawler IP, a second decision variable for judging whether the IP address belongs to at least one suspected crawler IP in the suspected crawler IP set, a third decision variable for judging whether the IP address belongs to a plurality of suspected crawler IPs in the suspected crawler IP set, and a fourth decision variable for representing the first second preset number of IP addresses in the suspected crawler IP set; A target function is constructed based on the dense feature vectors; the target function is: , The number of suspected crawler IPs is preset. For the fourth decision variable, The second preset quantity, For set The former The variance of each IP address; constructing a constraint condition set based on the dense feature vector; the constraint condition set includes a constraint condition for constraining the similarity and adjacency relationship between the suspected crawler IPs; solving the decision variable group, the objective function and the constraint condition set by a preset solver software to determine a plurality of suspected crawler IPs corresponding to the dense feature vector to obtain a suspected crawler IP set.

5. The method of claim 1 to 4, wherein, The hidden Markov model is constructed based on the time dimension for the suspected crawler IP set to determine the crawler IP in the suspected crawler IP set group by using the hidden Markov model, including: determining model three elements for constructing the hidden Markov model by a first preset algorithm; determining the crawler IP in the suspected crawler IP set group according to the model three elements by a second preset algorithm.

6. The crawler behavior detection method of claim 5, wherein, The model three elements for constructing the hidden Markov model are determined by a Baum-welch algorithm; And the crawler IP in the suspected crawler IP set group is determined according to the model three elements by a Viterbi algorithm. including: an original traffic data acquisition module for acquiring original traffic data and preprocessing the original traffic data to obtain key information tuples; 7. A crawler behavior detection apparatus characterized by comprising: a feature extraction module for determining a dense feature vector of each IP address in the original traffic data based on the key information tuples by using a Count Sketch algorithm and a principal component analysis method; the preprocessing is used to construct a standard data structure represented by an N-tuple; a spatial dimension modeling module for constructing a first mixed integer linear programming model based on a spatial dimension and determining a plurality of suspected crawler IPs corresponding to the dense feature vector by using the first mixed integer linear programming model to obtain a suspected crawler IP set; a time dimension modeling module for constructing a hidden Markov model based on a time dimension for the suspected crawler IP set to determine the crawler IP in the suspected crawler IP set group by using the hidden Markov model; The crawler behavior detection device is also used for: constructing the mixed integer linear programming model based on the spatial dimension again to obtain a second mixed integer linear programming model; ​ ​ The second mixed integer linear programming model is used to re-allocate IP addresses corresponding to the dense feature vectors to obtain an optimized target suspected crawler IP set.

8. An electronic device, comprising: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, the computer program is loaded and executed by the processor to implement the crawler behavior detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program for storing; wherein the computer program is executed by the processor to implement the crawler behavior detection method according to any one of claims 1 to 6.

Images