Multi-dimensional hybrid anti-crawler method, system, medium and equipment
Through a multi-dimensional hybrid anti-crawler method, combined with multiple algorithms and real-time update models, the problem of low anti-crawler detection in the existing technology is solved, accurate identification and effective defense of crawler behavior is achieved, and sensitive information in the web page is protected.
Patent Information
- Application Number
- CN202510378407.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-07-01
AI Technical Summary
The existing web anti-crawler technology has low detection of crawler behavior, resulting in sensitive information leaking user privacy.
The multi-dimensional hybrid anti-crawler method is adopted to collect user access logs, extract multi-dimensional features, and use a hybrid detection model (including isolated forest algorithm, local outlier factor algorithm and a type of support vector machine algorithm) to calculate the abnormal score, and trigger countermeasures when crawler behavior is detected, while the model is updated in real time and the threshold is adjusted to adapt to the new crawler behavior pattern.
It significantly improves the accuracy of crawler detection, reduces false positives and missed reports, effectively prevents crawlers from crawling data, enhances the adaptability and stability of the model, and protects the security of sensitive information.
Smart Images
Figure CN120234465A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and particularly to a multi-dimensional hybrid anti-crawler method, system, medium and device. Background Art
[0002] Crawler technology can automatically capture web page data. In some web pages containing sensitive information, once the data is captured by crawlers, the privacy of users will be leaked. The existing web page anti-crawler technology has a low detection rate for crawler behavior.
[0003] In view of this, it is necessary to provide a multi-dimensional hybrid anti-crawler method, system, medium and device. Summary of the Invention
[0004] A multi-dimensional hybrid anti-crawler method, system, medium and device provided by the present invention effectively solves the problem of the low existing web page anti-crawler technology.
[0005] The technical solution adopted by the present invention is as follows:
[0006] A multi-dimensional hybrid anti-crawler method includes the following steps:
[0007] S1. Collect user access logs and extract multi-dimensional features, including the number of requests, the number of unique paths, the request time distribution, the user agent type, the geographical location, and the device information;
[0008] S2. Standardize the extracted features and input them into a hybrid detection model, where the hybrid detection model includes the Isolation Forest algorithm, the Local Outlier Factor algorithm, and the One-Class Support Vector Machine algorithm;
[0009] S3. Calculate the anomaly score for each access behavior according to the output result of the hybrid detection model;
[0010] S4. When the anomaly score exceeds a preset threshold, it is determined as a crawler behavior, and countermeasures are triggered, including returning false data, restricting the access frequency, or blocking the IP;
[0011] S5. Update the hybrid detection model in real time and dynamically adjust the anomaly score threshold to adapt to new crawler behavior patterns.
[0012] Furthermore: The multi-dimensional features further include the variance of the request interval and the access order of the request paths.
[0013] Furthermore: The hybrid detection model combines the output results of the Isolation Forest algorithm, the Local Outlier Factor algorithm, and the One-Class Support Vector Machine algorithm through a weighted voting method.
[0014] Furthermore: The countermeasures further include setting up a honeypot page to trap crawlers and record their behaviors.
[0015] Furthermore, the method further includes encrypting and storing sensitive information during transmission, and adding differential privacy noise during data publication.
[0016] Furthermore, the hybrid detection model further includes a deep learning model, and the deep learning model is trained and optimized through the following steps: constructing a training set using historical access log data, and labeling normal users and crawler behaviors; inputting multi-dimensional features into a long short-term memory network (LSTM) to train the model to capture abnormal patterns in the time series; performing weighted fusion on the output of the LSTM model and the results of the isolation forest algorithm and the local outlier factor algorithm to generate a final abnormal score; and updating the parameters of the deep learning model in real time through an online learning mechanism to adapt to new crawler behavior patterns.
[0017] Furthermore, the multi-dimensional features further include user behavior pattern features, and the behavior pattern features are extracted through the following steps: analyzing the time series data of user accesses, and calculating the time interval distribution of requests; extracting the sequential pattern of the user access path to identify whether there is a fixed path access behavior; combining the user's geographical location and device information to determine whether there is abnormal access behavior; and combining the behavior pattern features with the request count and unique path count features and inputting them into the hybrid detection model for anomaly detection.
[0018] A multi-dimensional hybrid anti-crawler system
[0019] A log collection module for collecting user access logs;
[0020] A feature extraction module for extracting multi-dimensional features from access logs;
[0021] A hybrid detection module for calculating an abnormal score based on multi-dimensional features;
[0022] A countermeasure module for triggering countermeasures when crawler behavior is detected;
[0023] A model update module for updating the hybrid detection model in real time and dynamically adjusting the abnormal score threshold.
[0024] A computer-readable storage medium stores a computer program, and when the computer program is processed and executed, it implements the steps of the multi-dimensional hybrid anti-crawler method.
[0025] A computer device includes a processor, a communication interface, a memory, and a communication bus, and the processor, the communication interface, and the memory complete communication with each other through the communication bus: wherein:
[0026] The memory is used for storing a computer program;
[0027] The processor is configured to execute the steps of the multi-dimensional hybrid anti-crawler method by running the program stored in the memory.
[0028] Advantages of the invention:
[0029] 1. By integrating multi-dimensional features and hybrid models, it can describe user behavior more comprehensively, significantly improving the accuracy of crawler detection. Combining multiple algorithms such as Isolation Forest, Local Outlier Factor, and One-Class Support Vector Machine can avoid the limitations of a single algorithm, enhancing the adaptability and stability of the model; by updating the model in real-time and dynamically adjusting the anomaly score threshold, it can quickly adapt to new crawler behavior patterns, reducing false positives and false negatives. After detecting crawler behavior, anti-countermeasures are automatically triggered (such as returning false data, restricting access frequency, etc.) to effectively prevent the crawler from further scraping data.
[0030] 2. By setting up a honeypot page, the honeypot page can be used to set up a page with false sensitive information, and the crawler can be trapped by capturing the sensitive information, providing support for subsequent analysis and model optimization.
[0031] 3. Through deep learning, it can automatically adapt to the gradually improved crawler behavior patterns, improving the detection ability.
[0032] 4. Weightedly fuse the output of the deep learning model with the results of other algorithms to further improve the detection accuracy. Description of the drawings
[0033] Figure 1 It is a flowchart of the multi-dimensional hybrid anti-crawler method provided by the embodiments of the present application. Detailed implementation manners
[0034] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific implementation manners of the present invention will be given in conjunction with the accompanying drawings.
[0035] The first embodiment provided by the present application is a multi-dimensional hybrid anti-crawler method, including the following steps:
[0036] S1. Collect user access logs and extract multi-dimensional features, including the number of requests, the number of unique paths, the request time distribution, user agent type, geographical location, and device information;
[0037] S2. Standardize the extracted features and input them into a hybrid detection model, where the hybrid detection model includes an Isolation Forest algorithm, a Local Outlier Factor algorithm, and a One-Class Support Vector Machine algorithm;
[0038] S3. Calculate the anomaly score for each access behavior according to the output result of the hybrid detection model;
[0039] S4. When the anomaly score exceeds the preset threshold, it is determined as crawler behavior, and countermeasures are triggered, including returning false data, restricting the access frequency, or blocking the IP.
[0040] S5. Update the hybrid detection model in real time and dynamically adjust the anomaly score threshold to adapt to new crawler behavior patterns.
[0041] In the above design, by integrating multi-dimensional features and a hybrid model, the user behavior can be described more comprehensively, and the accuracy of crawler detection can be significantly improved. Combining multiple algorithms such as Isolation Forest, Local Outlier Factor, and One-Class Support Vector Machine can avoid the limitations of a single algorithm and enhance the adaptability and stability of the model. By updating the model in real time and dynamically adjusting the anomaly score threshold, it can quickly adapt to new crawler behavior patterns and reduce false positives and false negatives. After detecting crawler behavior, countermeasures (such as returning false data, restricting the access frequency, etc.) are automatically triggered to effectively prevent crawlers from further scraping data.
[0042] Specifically, the multi-dimensional features further include the variance of the request interval and the access order of the request path.
[0043] In the above design, by analyzing the time series data of user access and the path order pattern, more covert crawler behavior can be identified, and complex behavior patterns can be captured. Combining geographical location and device information can further distinguish normal users from crawlers, reduce misjudgment, and enhance the feature discrimination ability. Dynamic feature extraction enables the model to adapt to different types of crawler behavior and enhance the generalization performance.
[0044] Specifically, the hybrid detection model combines the output results of the Isolation Forest algorithm, the Local Outlier Factor algorithm, and the One-Class Support Vector Machine algorithm through a weighted voting method.
[0045] In the above design, the output of the deep learning model is weighted and fused with the results of other algorithms to further improve the detection accuracy.
[0046] Specifically, the countermeasures further include setting up a honeypot page to trap crawlers and record their behavior.
[0047] In the above design, the honeypot page can set a false sensitive information page, and use the crawler's scraping of sensitive information to trap the crawler, providing support for subsequent analysis and model optimization.
[0048] Specifically, the method further includes encrypting the storage and transmission of sensitive information and adding differential privacy noise when the data is published.
[0049] In the above design, by adding noise when the data is published, it is prevented that crawlers can scrape real sensitive information, further improving the security of the data.
[0050] Specifically, the hybrid detection model further includes a deep learning model, which is trained and optimized through the following steps: constructing a training set using historical access log data and annotating normal users and crawler behaviors; inputting multi-dimensional features into a long short-term memory network (LSTM) to train the model to capture abnormal patterns in the time series; performing weighted fusion on the output of the LSTM model and the results of the isolation forest algorithm and the local outlier factor algorithm to generate a final abnormal score; and updating the parameters of the deep learning model in real time through an online learning mechanism to adapt to new crawler behavior patterns.
[0051] In the above design, through deep learning, it is possible to automatically adapt to the gradually improved crawler behavior patterns and improve the detection ability.
[0052] Specifically, the multi-dimensional features further include user behavior pattern features, which are extracted through the following steps: analyzing the time series data of user accesses and calculating the time interval distribution of requests; extracting the sequential pattern of the user access path to identify whether there is a fixed path access behavior; combining the user's geographical location and device information to determine whether there is an abnormal access behavior; and combining the behavior pattern features with the request count and unique path count features and inputting them into the hybrid detection model for anomaly detection.
[0053] In the above design, by extracting user behavior pattern features, the dynamics of the crawler can be further controlled, and the web page anomaly detection ability can be improved.
[0054] The second embodiment provided by the present application is a multi-dimensional hybrid anti-crawler system, including
[0055] a log collection module for collecting user access logs;
[0056] a feature extraction module for extracting multi-dimensional features from the access logs;
[0057] a hybrid detection module for calculating an abnormal score based on the multi-dimensional features;
[0058] a countermeasure module for triggering countermeasures when crawler behavior is detected;
[0059] a model update module for updating the hybrid detection model in real time and dynamically adjusting the abnormal score threshold.
[0060] The third embodiment provided by this application is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is processed and executed, it implements the steps of the multi-dimensional hybrid anti-crawler method. In addition, the computer-readable storage medium in this embodiment can adopt any combination of one or more readable storage media. Among them, the readable storage medium includes systems, devices or components of electricity, light, electromagnetism, infrared or semiconductors, or any combination of the above.
[0061] The fourth embodiment provided by this application is a computer device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. Among them:
[0062] The memory is used to store a computer program;
[0063] The processor is used to execute the steps of the multi-dimensional hybrid anti-crawler method by running the program stored on the memory. As an implementation manner of the present invention, the communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.
[0064] As an implementation manner of the present invention, the communication interface is used for communication between the above terminal and other devices.
[0065] As an implementation manner of the present invention, the memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.
[0066] As an implementation manner of the present invention, the above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0067] The fifth embodiment provided by this application is a multi-dimensional hybrid anti-crawler method, which includes the following steps: S1. Collect user access logs and extract multi-dimensional features, including the number of requests, the number of unique paths, the request time distribution, the user agent type, the geographical location, and device information; S2. Standardize the extracted features and input them into the hybrid detection model, which includes the isolation forest algorithm, the local outlier factor algorithm, and the one-class support vector machine algorithm; S3. Calculate the anomaly score of each access behavior according to the output result of the hybrid detection model; S4. When the anomaly score exceeds the preset threshold, it is determined as a crawler behavior and countermeasures are triggered, including returning false data, restricting the access frequency, or blocking the IP.
[0068] S5. Update the hybrid detection model in real time and dynamically adjust the anomaly score threshold to adapt to new crawler behavior patterns. The multi-dimensional features further include the variance of the request interval and the access order of the request paths. The hybrid detection model combines the output results of the Isolation Forest algorithm, the Local Outlier Factor algorithm, and the One-Class Support Vector Machine algorithm through a weighted voting method. The countermeasures also include setting up a honeypot page to trap crawlers and record their behaviors. The method further includes encrypting the storage and transmission of sensitive information and adding differential privacy noise when the data is published. The hybrid detection model further includes a deep learning model, and the deep learning model is trained and optimized through the following steps: constructing a training set using historical access log data and annotating normal user and crawler behaviors; inputting the multi-dimensional features into a Long Short-Term Memory network (LSTM) to train the model to capture anomaly patterns in the time series; performing weighted fusion on the output of the LSTM model and the results of the Isolation Forest algorithm and the Local Outlier Factor algorithm to generate a final anomaly score; and updating the parameters of the deep learning model in real time through an online learning mechanism to adapt to new crawler behavior patterns. The multi-dimensional features further include user behavior pattern features, and the behavior pattern features are extracted through the following steps: analyzing the time series data of user accesses, calculating the distribution of request time intervals; extracting the sequential pattern of the user access path and identifying whether there is a fixed path access behavior; combining the user's geographical location and device information to determine whether there is an abnormal access behavior; and combining the behavior pattern features with the request count and unique path count features and inputting them into the hybrid detection model for anomaly detection.
[0069] In the above design, it is possible to effectively detect web crawler behaviors and effectively prevent sensitive information in web data from being grabbed by illegal crawler behaviors.
[0070] For further details, it should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.
Claims
1. A multi-dimensional hybrid anti-crawler method, characterized in that: The following steps are involved: S1. Collect user access logs and extract multi-dimensional features, including the number of requests, the number of unique paths, the request time distribution, the user agent type, the geographic location, and the device information; S2, standardizing the extracted features and inputting them into a hybrid detection model, wherein the hybrid detection model includes an isolation forest algorithm, a local outlier factor algorithm, and a type of support vector machine algorithm; S3. Calculate the abnormal score of each access behavior based on the output results of the hybrid detection model; S4. When the anomaly score exceeds the preset threshold, it is determined to be crawler behavior and triggers countermeasures, including returning false data, limiting access frequency or banning IP; S5. Update the hybrid detection model in real time and dynamically adjust the anomaly score threshold to adapt to new crawler behavior patterns.
2. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The multi-dimensional features also include the variance of the request interval and the access sequence of the request path.
3. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The hybrid detection model combines the output results of the isolation forest algorithm, the local outlier factor algorithm and a type of support vector machine algorithm through weighted voting.
4. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The countermeasures also include setting up a honeypot page to trap crawlers and record their behavior.
5. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The method also includes encrypting the storage and transmission of sensitive information and adding differential privacy noise when publishing data.
6. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The hybrid detection model also includes a deep learning model, which is trained and optimized through the following steps: using historical access log data to build a training set and annotate normal user and crawler behaviors; inputting multidimensional features into a long short-term memory network (LSTM) to train the model to capture abnormal patterns in time series; weightedly fusing the output of the LSTM model with the results of the isolation forest algorithm and the local outlier factor algorithm to generate a final anomaly score; and updating the parameters of the deep learning model in real time through an online learning mechanism to adapt to new crawler behavior patterns.
7. The multi-dimensional hybrid anti-crawler method according to claim 1, characterized in that: The multidimensional features also include user behavior pattern features, which are extracted by: analyzing the time series data of user accesses and calculating the time interval distribution of requests; Extract the sequential pattern of the user's access path to identify whether there is fixed path access behavior; combine the user's geographic location and device information to determine whether there is abnormal access behavior; combine the behavior pattern characteristics with the number of requests and the number of unique paths, and input them into the hybrid detection model for anomaly detection.
8. A multi-dimensional hybrid anti-crawler system, characterized by: Log collection module, used to collect user access logs; Feature extraction module, used to extract multi-dimensional features from access logs; A hybrid detection module for calculating anomaly scores based on multi-dimensional features; The countermeasure module is used to trigger countermeasures when crawler behavior is detected; Model update module, which is used to update the hybrid detection model in real time and dynamically adjust the anomaly score threshold.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is processed and executed, the steps of the multi-dimensional hybrid anti-crawler method according to any one of claims 1 to 7 are implemented.
10. A computer device, characterized in that: The method comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus: The memory is used to store computer programs; The processor is used to execute the steps of the multi-dimensional hybrid anti-crawler method according to any one of claims 1 to 7 by running the program stored in the memory.