Intelligent anti-crawler device and method based on streaming computing and unsupervised learning

Abstract

The invention discloses an intelligent anti-crawler device and method based on streaming computing and unsupervised learning, and belongs to the technical field of computers.The device comprises a data collecting and buffering module used for collecting user access logs in real time and buffering log data to a message queue; the real-time processing and intelligent detection module is used for consuming the log data in the message queue and carrying out anomaly detection to obtain an anomaly detection result; and the intelligent decision module is used for generating a disposal instruction according to the anomaly detection result. Streaming calculation, feature engineering, unsupervised learning, a dynamic strategy engine and a micro-service gateway are organically fused to form a highly automatic and adaptive closed-loop anti-crawler device, and a real-time intelligent detection module based on unsupervised learning integrates and applies various unsupervised learning algorithms to a streaming data environment. Real-time and accurate discovery of unknown behaviors is realized, and dependence on labeled data is eliminated.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to an intelligent anti-crawler device and method based on streaming computing and unsupervised learning. Background Technology

[0002] With the rapid development of internet technology, malicious web crawlers pose a serious threat to the security of websites and web applications, leading to problems such as data leaks, excessive consumption of server resources, and abuse of business interfaces. Traditional anti-crawler technologies mainly have the following shortcomings: Lagging rules: Relying on predefined rule bases (such as dynamic pages, IP blacklists, and User-Agent filtering) makes it difficult to cope with unknown and rapidly evolving crawler behaviors, resulting in passive defense.

[0003] Data dependency: Detection models based on supervised learning require a large amount of labeled "web crawler" and "normal user" data, which is costly and cannot identify new types of attacks.

[0004] Limited response methods: Usually, only simple measures such as "banning" are used, which can easily affect normal users and lacks refined control capabilities.

[0005] System silos: The detection, analysis, and handling processes are independent of each other, failing to form a closed loop and thus unable to achieve adaptive defense.

[0006] Therefore, applications urgently need an automated system solution that can identify and deal with malicious crawlers in real time, accurately and adaptively, with minimal impact on normal users. Summary of the Invention

[0007] To address the aforementioned problems, this application provides an intelligent anti-scraping device based on streaming computing and unsupervised learning, the device comprising: The data acquisition and buffering module is used to collect user access logs in real time and buffer the log data to a message queue. The real-time processing and intelligent detection module is used to consume log data in the message queue, perform anomaly detection, and obtain anomaly detection results. The intelligent decision-making module is used to generate handling instructions based on anomaly detection results; The dynamic handling execution module is used to execute handling instructions and perform fine-grained control over abnormal traffic.

[0008] According to some embodiments of this disclosure, the data acquisition and buffering module includes: The data collection unit is used to collect newly added logs in real time. A format standardization unit is used to perform format standardization processing on unstructured logs; wherein, the format standardization processing includes converting unstructured logs into a predefined structured data format; The message queue push unit is used to push standardized log data to a high-performance message queue cluster.

[0009] According to some embodiments of this disclosure, the real-time processing and intelligent detection module includes: The data consumption unit is used to read log data streams; The data parsing and cleaning unit is used to parse log data in the log data stream, extract target fields, and perform log data cleaning logic. The feature engineering unit is used to perform multi-dimensional behavioral feature calculations on the cleaned log data. An anomaly detection unit is used to input multi-dimensional behavioral features into an unsupervised machine learning model to obtain anomaly detection results; wherein, the anomaly detection results include user identifier, anomaly score, anomaly type, and timestamp; The output unit is used to output the anomaly detection results as a new data stream.

[0010] According to some embodiments of this disclosure, the multi-dimensional behavioral characteristics include time series characteristics, resource access characteristics, and environmental identification characteristics; The time series features include the mean of the request interval, the median of the request interval, and the standard deviation of the request interval; The resource access characteristics include the number of unique URIs, the Shannon entropy of the URI access distribution, and the entropy of the citation source distribution; The environmental identification features include the number of different IPs used during the session, the entropy value of the IP distribution, and the total number of requests.

[0011] According to some embodiments of this disclosure, the unsupervised machine learning model includes at least two of the following: the Isolation Forest algorithm, the DBSCAN clustering algorithm, the K-Means clustering algorithm, and the Gaussian mixture model. The isolated forest algorithm performs anomaly detection based on path length; The DBSCAN clustering algorithm identifies outliers based on density. The K-Means clustering algorithm performs clustering analysis based on distance; The Gaussian mixture model is based on a probability model for cluster analysis.

[0012] According to some embodiments of this disclosure, the unsupervised machine learning model uses a voting mechanism to make the final decision, and when at least two algorithms determine that a user is abnormal, the user is marked as an abnormal user.

[0013] According to some embodiments of this disclosure, the intelligent decision-making module includes: The event receiving unit is used to receive anomaly detection results; The policy matching unit is used to load policy rules from the dynamic policy rule base and perform policy matching based on the anomaly detection results; The instruction generation and issuance unit is used to generate disposal instructions based on the policy matching results and store the disposal instructions in the cache database.

[0014] According to some embodiments of this disclosure, the content of the disposal instruction includes a disposal action, a disposal target, action parameters, and an instruction validity period. The types of processing instructions include human-machine verification, flow restriction, blocking, observation, and release.

[0015] According to some embodiments of this disclosure, the dynamic processing execution module includes: The instruction monitoring unit is used to query and process instructions through a global filter; The instruction execution unit is used to execute corresponding processing actions according to the type of processing instruction; wherein the processing actions include at least one of returning to the verification code page, limiting the request rate, and directly rejecting the request.

[0016] This application also provides an intelligent anti-scraping method based on streaming computing and unsupervised learning, the method comprising: Collect user access logs in real time and buffer the log data to a message queue; Consume log data from the message queue, perform anomaly detection, and obtain the anomaly detection results; Generate handling instructions based on anomaly detection results; Execute handling instructions and implement refined control over abnormal traffic.

[0017] Compared with the prior art, this application has the following advantages: (1) This application organically integrates streaming computing, feature engineering, unsupervised learning, dynamic policy engine, and microservice gateway through an innovative process construction, forming for the first time a highly automated and adaptive closed-loop anti-crawler device. Among them, the real-time intelligent detection module based on unsupervised learning creatively integrates multiple unsupervised learning algorithms and applies them to the streaming data environment, realizing real-time and accurate detection of unknown behaviors and getting rid of dependence on labeled data. The intelligent hierarchical handling mechanism proposes a dynamic decision model based on the combination of anomaly scores and types, realizing a refined hierarchical response from "observation" to "blocking", which is rare in the existing technology. In addition, the decision system and API gateway are deeply integrated at the control level, realizing the lossless and efficient execution of network security policies at the traffic entry point.

[0018] (2) This application precisely addresses two core pain points in the current Web security field: "lagging passive defense rules" and "supervised learning relying on annotation." By constructing a real-time, intelligent, and automated closed-loop anti-crawler system, it achieves a paradigm shift from "passive response" to "proactive perception and precise handling." This patent is not a single algorithm application, but a systematic engineering innovation. It involves the deep integration of multiple technical fields such as streaming computing, feature engineering, machine learning, and microservice governance. This technology can be widely applied to any industry troubled by crawler problems, such as finance, e-commerce, online ticketing, content platforms, and API service providers. It has a huge market potential and the potential to become an industry standard solution.

[0019] (3) Independent "Application Security Intelligent Management Platform" SaaS products or deployment software can be developed directly based on this application to provide services to customers. For each company, applying this product can significantly reduce business losses and server costs caused by web crawlers, enhance the platform's security image and user experience, and constitute part of its core technological competitiveness.

[0020] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A diagram of an intelligent anti-crawler device based on streaming computing and unsupervised learning according to an embodiment of this application is shown. Figure 2 A flowchart of a smart anti-crawler device based on streaming computing and unsupervised learning according to an embodiment of this application is shown. Figure 3 A diagram illustrating an intelligent anti-crawler method based on streaming computing and unsupervised learning according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] This application aims to address the aforementioned deficiencies in the prior art, and mainly solves the following technical problems: How to achieve real-time and accurate identification of unknown and variant crawler behaviors, and reduce reliance on pre-labeled data.

[0025] How to build an automated closed-loop system from perception and decision-making to action, breaking down system silos.

[0026] How to achieve tiered and precise handling and response to avoid accidental damage to business operations caused by a single strategy.

[0027] Explanation of technical terms User-Agent: User agent is a special string that identifies the client's hardware and software configuration to the server, including information such as operating system, browser version, and rendering engine; Supervised learning: Supervised learning is the process of adjusting the parameters of a classifier using a set of samples of known classes to achieve the required performance. It is also called supervised training or teacher-led learning. Streaming computing: a real-time processing paradigm for unbounded data streams. Unlike traditional batch processing (which processes bounded datasets), streaming computing engines process data immediately after it is generated, enabling low-latency (milliseconds to seconds) insights and responses. Unsupervised learning: A class of machine learning algorithms designed to discover inherent, unlabeled patterns in data. It does not rely on pre-labeled training data, but rather identifies patterns and anomalies by analyzing the distribution, density, or structure of the data itself. Feature engineering: The process of constructing features from raw data that effectively characterize business problems. These features serve as input to machine learning models, and their quality directly determines the upper limit of model performance. Shannon entropy: A metric in information theory used to quantify the uncertainty of random variables. In this invention, it is used to measure the randomness and diversity of user behavior (such as the URLs visited). Isolation Forest: An unsupervised learning algorithm specifically designed for anomaly detection. Its core idea is that outliers are sparse and distinct, therefore, by randomly selecting features and splitting values, they can be "isolated" from normal data using a shorter path. DBSCAN: A density-based spatial clustering algorithm. It defines a cluster as the largest set of density-connected points and can identify noise points that do not belong to any cluster as anomalies. K-Means: A clustering analysis algorithm that discovers similarities in a dataset by grouping objects; it is one of the popular unsupervised machine learning (ML) algorithms. Gaussian Mixture Model: A clustering algorithm based on a probabilistic model. It assumes that all data points are generated by a mixture of multiple Gaussian distributions and fits the data by estimating the parameters of each Gaussian distribution; SaaS: Software as a Service, refers to a new form of software service that users can obtain. It involves obtaining the required software functions directly from a specialized provider via the internet according to a certain Service Level Agreement (SLA).

[0028] like Figure 1 As shown, this application provides an intelligent anti-scraping device based on streaming computing and unsupervised learning, the device comprising: The data acquisition and buffering module is used to collect user access logs in real time and buffer the log data to a message queue. The real-time processing and intelligent detection module is used to consume log data in the message queue, perform anomaly detection, and obtain anomaly detection results. The intelligent decision-making module is used to generate handling instructions based on anomaly detection results; The dynamic handling execution module is used to execute handling instructions and perform fine-grained control over abnormal traffic.

[0029] Specifically, the data acquisition and buffering module includes: The data collection unit is used to collect newly added logs in real time. A format standardization unit is used to perform format standardization processing on unstructured logs; wherein, the format standardization processing includes converting unstructured logs into a predefined structured data format; The message queue push unit is used to push standardized log data to a high-performance message queue cluster.

[0030] Specifically, the real-time processing and intelligent detection module includes: The data consumption unit is used to read log data streams; The data parsing and cleaning unit is used to parse log data in the log data stream, extract target fields, and perform log data cleaning logic. The feature engineering unit is used to perform multi-dimensional behavioral feature calculations on the cleaned log data. An anomaly detection unit is used to input multi-dimensional behavioral features into an unsupervised machine learning model to obtain anomaly detection results; wherein, the anomaly detection results include user identifier, anomaly score, anomaly type, and timestamp; The output unit is used to output the anomaly detection results as a new data stream.

[0031] Specifically, the multi-dimensional behavioral characteristics include time series characteristics, resource access characteristics, and environmental identification characteristics; The time series features include the mean of the request interval, the median of the request interval, and the standard deviation of the request interval; The resource access characteristics include the number of unique URIs, the Shannon entropy of the URI access distribution, and the entropy of the citation source distribution; The environmental identification features include the number of different IPs used during the session, the entropy value of the IP distribution, and the total number of requests.

[0032] Specifically, the unsupervised machine learning model includes at least two of the following: the Isolation Forest algorithm, the DBSCAN clustering algorithm, the K-Means clustering algorithm, and the Gaussian mixture model. The isolated forest algorithm performs anomaly detection based on path length; The DBSCAN clustering algorithm identifies outliers based on density. The K-Means clustering algorithm performs clustering analysis based on distance; The Gaussian mixture model is based on a probability model for cluster analysis.

[0033] Specifically, the unsupervised machine learning model uses a voting mechanism to make the final decision, and when at least two algorithms determine that a user is abnormal, the user is marked as an abnormal user.

[0034] Specifically, the intelligent decision-making module includes: The event receiving unit is used to receive anomaly detection results; The policy matching unit is used to load policy rules from the dynamic policy rule base and perform policy matching based on the anomaly detection results; The instruction generation and issuance unit is used to generate disposal instructions based on the policy matching results and store the disposal instructions in the cache database.

[0035] Specifically, the content of the disposal instruction includes the disposal action, the disposal target, the action parameters, and the instruction validity period. The types of processing instructions include human-machine verification, flow restriction, blocking, observation, and release.

[0036] Specifically, the dynamic processing execution module includes: The instruction monitoring unit is used to query and process instructions through a global filter; The instruction execution unit is used to execute corresponding processing actions according to the type of processing instruction; wherein the processing actions include at least one of returning to the verification code page, limiting the request rate, and directly rejecting the request.

[0037] The detailed technical solution for this device is as follows: The data acquisition and buffering module is used to collect user access logs from distributed application servers in real time and push them to a high-performance message queue (such as Kafka) for buffering and decoupling. Its core task is to transform unstructured log text into structured data, achieving high availability and scalability for log acquisition, transmission, and computation.

[0038] Real-time processing and intelligent detection module: Built on a streaming computing engine (such as Apache Flink), it consumes log streams from the message queue and performs the following operations: Real-time feature engineering: Calculates multi-dimensional behavioral features in real time at the user session level, including time-series features (mean and standard deviation of request intervals), resource access diversity features (URI entropy and Referrer entropy), and environmental identification features (IP entropy).

[0039] Unsupervised anomaly detection: An integrated unsupervised learning algorithm is used, including but not limited to Isolation Forest, DBSCAN clustering, K-Means clustering and Gaussian mixture model, to perform parallel inference on real-time feature vectors and output anomaly probability scores and anomaly type labels.

[0040] Intelligent Decision Module: This module includes a real-time decision engine and a dynamically configurable policy rule base. It receives anomaly detection results and generates precise handling instructions (such as observation, human verification, rate limiting, blocking, and allowing access) based on preset, hierarchical policies that combine anomaly scores and anomaly types.

[0041] Dynamic Handling Execution Module: Deeply integrated with the API gateway, this module receives and executes instructions issued by the decision engine. At the network traffic entry point, it performs corresponding handling actions on abnormal traffic, achieving non-destructive intervention.

[0042] like Figure 2 As shown, the implementation principle of this device is as follows: Data Acquisition and Buffering Module Step 1: Agent collection: Deploy a lightweight log collection agent (such as Filebeat) on each server to monitor and capture new logs in real time.

[0043] Step 2, Format Standardization: Convert unstructured log text into a predefined, machine-readable structured data format (JSON format), including key fields such as IP, User-Agent, URL, and timestamp.

[0044] Step 3: Push to message queue: The collection agent continuously pushes the standardized log data as messages to the specified Topic in the Apache Kafka cluster.

[0045] Real-time processing and intelligent detection module Step 1: Data Consumption: An Alink stream processing job running on an Apache Flink cluster acts as a consumer to subscribe to and read log data streams from Kafka.

[0046] Step 2: Data Parsing and Cleaning: Parse the JSON-formatted log messages and extract key fields. Execute data cleaning logic, such as filtering out health checks, static resources, and obviously invalid requests.

[0047] Step 3, Feature Engineering: Within each session window, multi-dimensional behavioral features are calculated in real time, including time series features, resource access features, and environmental identification features.

[0048] Step 4, Model Inference and Anomaly Detection: The feature vectors calculated in real time are input into a series of pre-trained unsupervised machine learning models. The models work in parallel and output their respective detection results.

[0049] Step 5: Output the detection results: Write the final anomaly detection results (including user ID, anomaly score, anomaly type, and timestamp) as a new data stream to another designated topic in Kafka for the decision-making module to consume.

[0050] Intelligent decision-making module Step 1: Event Reception: The decision engine consumes the abnormal result topics output by the detection module in Kafka in real time.

[0051] Step 2, Policy Matching: The decision engine loads the currently effective policy rules from the dynamic policy rule library, with the condition part based on a combination of anomaly_score and anomaly_type.

[0052] Step 3, Command Generation and Issuance: Generate structured handling commands. The command content includes: action (handling action), target (handling target, such as IP address or User ID), params (action parameters, such as rate limiting), TTL (command validity period), etc. The commands are written to a cache database (such as Redis) in the form of key-value pairs.

[0053] Dynamic processing execution module Step 1: Command Listening: By using the API gateway deployed at the traffic entry point and the configured global filter, query the Redis cache to see if there is a corresponding handling command.

[0054] Step 2, Instruction Execution: If no instruction is given, the request passes through the gateway normally and is routed to the backend business service; if an instruction is given, the gateway immediately interrupts the default routing process and executes the handling action defined by the instruction.

[0055] Example The following section will explain the contents of this device in conjunction with the specific implementation process.

[0056] 1. Data Acquisition and Buffering Deploy log collection agents such as Filebeat on the application server to monitor log files (such as Nginx Access Log and application custom Log) in real time, parse each log (for example, parse it into JSON format, including key fields such as IP, Timestamp, URL, User-Agent, and SessionID), and push the structured (JSON format) access logs to the specified Topic of the Kafka cluster in real time.

[0057] 2. Real-time processing and intelligent detection 2.1 Data Analysis and Feature Engineering Run an Alink streaming job to consume log data from Kafka. The job first parses and cleans the data, then divides the session window by user_id or user_agent. Within each window, it calculates the following 9-dimensional behavioral feature vector in real time: 2.1.1 Time Series Characteristics These characteristics characterize user access frequency patterns and are a powerful method for identifying automated scripts (which typically have a fixed frequency).

[0058] (1) interval_mean: The arithmetic mean of the time intervals between consecutive requests. The mean of a crawler is usually significantly smaller than that of a human user.

[0059] (2) interval_median: The median of the time interval. It is not sensitive to extreme values ​​and can better reflect the typical interval.

[0060] (3) interval_std: standard deviation of the time interval. The access interval of human users is more random (large standard deviation), while that of web crawlers is very regular (standard deviation close to zero).

[0061] 2.1.2. Diverse Characteristics of Resource Access These features differentiate intent by assessing the breadth and randomness of user access behavior.

[0062] (1) uri_count: The number of unique URIs accessed. For example, scanners usually access a large number of different URIs, while crawlers typically access specific interfaces frequently.

[0063] (2) uri_entropy: Shannon entropy of URI access distribution. The higher the entropy value, the more uniform the access distribution, and the more the behavior is like human random browsing; a low entropy value may indicate purposeful repeated access or scanning.

[0064] (3) referer_entropy: the entropy of the source distribution of the reference, similar in meaning to uri_entropy, used to determine whether the navigation path is natural.

[0065] 2.1.3 Environmental Labeling Characteristics These features are used to identify web crawlers that use tools to conceal their behavior.

[0066] (1) ip_count: The number of different IPs used during the session. A value greater than 1 strongly suggests that a proxy IP pool was used.

[0067] (2) ip_entropy: The entropy value of IP distribution. Even if the number of IPs is not large, a high entropy value indicates that the IP switching is irregular, which is consistent with the characteristics of a proxy pool.

[0068] (3) req_count: The total number of requests during the session, which is a basic indicator for measuring the intensity of activity.

[0069] 2.2 Intelligent Detection A pre-trained unsupervised learning model is loaded, and feature vectors are scored, outputting anomaly_score and anomaly_type. As shown in Table 1, this unsupervised learning model integrates multiple algorithms (DBSCAN, K-Means, GMM, and Isolation Forest), and through a voting mechanism, greatly enhances the system's reliability in real-world complex environments. Ensemble rule: For a user sample, if at least two of the three algorithms (DBSCAN, K-Means, GMM) classify it as an anomaly (i.e., ensemble_anomaly >= 2), then it is finally labeled as an anomalous user (final_anomaly = 1). Clustering may discover small, dense clusters of coordinated attacks, while Isolation Forest may be better at identifying isolated anomalies in a globally significant sense.

[0070] Table 1 Unsupervised learning model algorithms

[0071] 3. Intelligent decision-making The policy rule base uses configurable rule sets, supports dynamic loading, and takes effect without requiring a service restart. The decision engine queries the policy rule base, for example, matching the rule: "If anomaly_score > 0.9, then execute action=block". It generates a disposal instruction (e.g., {"action": "rate_limit", "target": "ip:192.168.1.1", "params": {"rate": "10r / m"}}) and writes it to the Redis cache. Rules are defined based on a combination of anomaly_score and anomaly_type, as shown in Table 2. Table 2 Strategy Rule Base

[0072] 4. Dynamic handling Before processing each inbound request, the API gateway (such as Spring Cloud Gateway or Nginx+Lua) checks Redis for any instructions on how to handle the request source (IP, user_id, etc.). If instructions exist, they are executed immediately: such as returning a CAPTCHA page (human verification), limiting the request rate (rate limiting), or directly rejecting the request (blocking). (1) Human-machine verification: The request is redirected to the verification code service page, or the verification code is returned directly in the response. After the user is verified, the verification code service notifies the gateway to remove the instruction.

[0073] (2) Rate limiting: Using algorithms such as token bucket, the number of requests to the target within a unit of time is strictly limited. Requests exceeding this limit will be delayed or returned with a 429 status code.

[0074] (3) Block: Directly reject the request and return a 403 status code.

[0075] (4) Observation: Low-risk abnormalities are not subject to actual intervention, but are recorded in the log.

[0076] (5) Allow: Allow IPs or accounts in the whitelist directly.

[0077] like Figure 3 As shown, this application also provides an intelligent anti-crawler method based on streaming computing and unsupervised learning, the method comprising: Collect user access logs in real time and buffer the log data to a message queue; Consume log data from the message queue, perform anomaly detection, and obtain the anomaly detection results; Generate handling instructions based on anomaly detection results; Execute handling instructions and implement refined control over abnormal traffic.

[0078] This application, through an innovative process construction, organically integrates streaming computing, feature engineering, unsupervised learning, a dynamic policy engine, and a microservice gateway, forming for the first time a highly automated and adaptive closed-loop anti-crawler system. The real-time intelligent detection module based on unsupervised learning creatively integrates multiple unsupervised learning algorithms for use in a streaming data environment, achieving real-time and accurate detection of unknown behaviors and eliminating reliance on labeled data. The intelligent hierarchical handling mechanism proposes a dynamic decision-making model based on anomaly scores and type combinations, achieving a refined hierarchical response from "observation" to "blocking," which is rare in existing technologies. Furthermore, the deep integration of the decision-making system and the API gateway at the control level enables lossless and efficient execution of network security policies at the traffic entry point. These are the innovative points of this application in engineering practice. This application has the following value: (1) Possesses significant technological leadership and the ability to solve industry pain points: This application precisely addresses two core pain points in the current Web security field: "lagging passive defense rules" and "supervised learning relying on annotation." By constructing a real-time, intelligent, and automated closed-loop anti-crawler system, it achieves a paradigm shift from "passive response" to "proactive perception and precise handling." This application is not a single algorithm application, but a systematic engineering innovation. It involves the deep integration of multiple technical fields such as streaming computing, feature engineering, machine learning, and microservice governance. This technology can be widely applied to any industry troubled by crawler problems, such as finance, e-commerce, online ticketing, content platforms, and API service providers. It has a huge market potential and the potential to become an industry standard solution.

[0079] (2) Clear productization path: Independent "Application Security Intelligent Management Platform" SaaS products or deployment software can be developed directly based on this application to provide services to customers. For each company, applying this product can significantly reduce business losses and server costs caused by web crawlers, enhance the platform's security image and user experience, and constitute part of its core technological competitiveness.

[0080] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An intelligent anti-crawler device based on streaming computing and unsupervised learning, characterized in that, The device includes: The data acquisition and buffering module is used to collect user access logs in real time and buffer the log data to a message queue. The real-time processing and intelligent detection module is used to consume log data in the message queue, perform anomaly detection, and obtain anomaly detection results. The intelligent decision-making module is used to generate handling instructions based on anomaly detection results; The dynamic handling execution module is used to execute handling instructions and perform fine-grained control over abnormal traffic.

2. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 1, characterized in that, The data acquisition and buffering module includes: The data collection unit is used to collect newly added logs in real time. A format standardization unit is used to perform format standardization processing on unstructured logs; wherein, the format standardization processing includes converting unstructured logs into a predefined structured data format; The message queue push unit is used to push standardized log data to a high-performance message queue cluster.

3. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 1, characterized in that, The real-time processing and intelligent detection module includes: The data consumption unit is used to read log data streams; The data parsing and cleaning unit is used to parse log data in the log data stream, extract target fields, and perform log data cleaning logic. The feature engineering unit is used to perform multi-dimensional behavioral feature calculations on the cleaned log data. An anomaly detection unit is used to input multi-dimensional behavioral features into an unsupervised machine learning model to obtain anomaly detection results; wherein, the anomaly detection results include user identifier, anomaly score, anomaly type, and timestamp; The output unit is used to output the anomaly detection results as a new data stream.

4. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 3, characterized in that, The multi-dimensional behavioral characteristics include time series characteristics, resource access characteristics, and environmental identification characteristics; The time series features include the mean of the request interval, the median of the request interval, and the standard deviation of the request interval; The resource access characteristics include the number of unique URIs, the Shannon entropy of the URI access distribution, and the entropy of the citation source distribution; The environmental identification features include the number of different IPs used during the session, the entropy value of the IP distribution, and the total number of requests.

5. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 3, characterized in that, The unsupervised machine learning model includes at least two of the following: Isolation Forest algorithm, DBSCAN clustering algorithm, K-Means clustering algorithm, and Gaussian mixture model; The isolated forest algorithm performs anomaly detection based on path length; The DBSCAN clustering algorithm identifies outliers based on density. The K-Means clustering algorithm performs clustering analysis based on distance; The Gaussian mixture model is based on a probability model for cluster analysis.

6. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 5, characterized in that, The unsupervised machine learning model uses a voting mechanism for final decision-making. When at least two algorithms determine that a user is abnormal, the user is marked as an abnormal user.

7. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 1, characterized in that, The intelligent decision-making module includes: The event receiving unit is used to receive anomaly detection results; The policy matching unit is used to load policy rules from the dynamic policy rule base and perform policy matching based on the anomaly detection results; The instruction generation and issuance unit is used to generate disposal instructions based on the policy matching results and store the disposal instructions in the cache database.

8. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 7, characterized in that, The disposal instruction includes the disposal action, disposal target, action parameters, and instruction validity period. The types of processing instructions include human-machine verification, flow restriction, blocking, observation, and release.

9. The intelligent anti-crawler device based on streaming computing and unsupervised learning according to claim 8, characterized in that, The dynamic processing execution module includes: The instruction monitoring unit is used to query and process instructions through a global filter; The instruction execution unit is used to execute corresponding processing actions according to the type of processing instruction; wherein the processing actions include at least one of returning to the verification code page, limiting the request rate, and directly rejecting the request.

10. An intelligent anti-crawler method based on streaming computing and unsupervised learning, characterized in that, The method includes: Collect user access logs in real time and buffer the log data to a message queue; Consume log data from the message queue, perform anomaly detection, and obtain the anomaly detection results; Generate handling instructions based on anomaly detection results; Execute handling instructions and implement refined control over abnormal traffic.

Images