Intelligent malicious crawler prevention defense method based on Java

Abstract

The invention provides an intelligent malicious crawler defense method based on Java. The method comprises the following steps: collecting multi-dimensional metadata of a user access request; performing structured processing and feature engineering on the multi-dimensional metadata to generate feature vectors for behavior analysis; constructing and applying a double-layer recognition mechanism fusing a rule engine and a machine learning model, and calculating a probability value P of the user access behavior belonging to a malicious crawler based on the feature vector; setting an initial judgment threshold T according to statistical analysis of historical data, and dynamically optimizing and adjusting the judgment threshold T based on a misjudgment rate and a missed judgment rate; and comparing the probability value P with the current judgment threshold T, and continuously optimizing the machine learning model and the feature weight based on the feedback of the defense result. The problems that an existing crawler prevention method is poor in protection effect, high in misjudgment rate, poor in adaptability and the like are solved, accurate recognition and efficient interception of malicious crawlers are achieved, and meanwhile good access experience of normal users is guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of computer network security technology, specifically relating to a Java-based intelligent anti-malicious web crawler defense method. Background Technology

[0002] In the internet age, web crawlers play a vital role in information acquisition, but the proliferation of malicious crawlers has brought numerous problems to websites. Malicious crawlers use high-frequency requests and camouflage to scrape large amounts of data, consuming server resources, affecting website performance, and even leading to data leaks and intellectual property infringement.

[0003] Existing anti-scraping methods have many shortcomings: static verification mechanisms are easily cracked; interception strategies based on single features have a high false positive rate; they lack dynamic adaptability and are difficult to cope with the continuous upgrading of crawler technologies; and most methods have poor compatibility with the Java ecosystem, making integration difficult. Therefore, there is a need for a Java-based anti-malicious crawling method with dynamic recognition capabilities, a low false positive rate, and easy integration. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a Java-based intelligent anti-malicious web crawler defense method, which aims to solve the problems mentioned in the background art, such as poor protection effect, high false positive rate and weak adaptability of existing anti-web crawler methods.

[0005] To achieve the above objectives, one or more embodiments of this application provide a Java-based intelligent anti-malicious web crawler defense method. Includes the following steps: S1: Collect multi-dimensional metadata of user access requests; S2: Perform structured processing and feature engineering on the multi-dimensional metadata to generate feature vectors for behavior analysis; S3: Construct and apply a two-layer identification mechanism that integrates a rule engine and a machine learning model, and calculate the probability value P that the user access behavior belongs to a malicious crawler based on the feature vector; S4: Set an initial judgment threshold T based on historical data statistical analysis, and dynamically optimize and adjust the judgment threshold T based on the false judgment rate and the false negative rate; S5: Compare the probability value P with the current judgment threshold T, execute the hierarchical defense strategy based on the comparison result, and continuously optimize the machine learning model and feature weights based on the feedback of the defense result.

[0006] Based on the above technical solution of the present invention, the following improvements can also be made: Optionally, in step S1, the multi-dimensional metadata includes: HTTP request header information, access time series data, user behavior trajectory data, and network feature data.

[0007] Optionally, step S2 includes: S21: Perform a structured transformation on the metadata to generate structured data containing fixed fields; S22: Perform Z-Score standardization on numerical features to eliminate the influence of different orders of magnitude; S23: Perform one-hot encoding on categorical features; S24: Extract the temporal features of access behavior; S25: Finally, the data processed by steps S21 to S24 are combined into a feature vector representing user access behavior.

[0008] Optionally, the rule engine in step S3 has at least one of the following preset rules: Rule 1: Monitor the number of requests made by a single IP address within a unit of time. If the number exceeds a preset threshold, it will be marked as suspicious. Rule 2: Check the User-Agent field in the request header. If it matches the built-in crawler feature library, it is determined to have crawler features. Rule 3: Monitor the frequency of IP addresses triggering 404 errors within a short period of time. If the frequency exceeds a preset threshold, it will be identified as abnormal behavior.

[0009] Optionally, the rule engine can perform the following operations: record the IP address information that triggers the rule to a blacklist or an abnormal behavior alarm table, and periodically push the alarm information to the person in charge.

[0010] Optionally, the machine learning model described in step S3 is a classification model, which outputs a probability value P through the following steps: The model is trained using feature vectors from historical access data; Input the real-time feature vector into the trained model to obtain the basic probability value P that the access behavior belongs to a malicious crawler. If the P value is within a preset fuzzy range, the rule engine is triggered to perform a second verification, and the P value is corrected based on the verification result.

[0011] Optionally, after obtaining the base probability value P, the final probability value P is calculated further through the following steps: The basic probability value P is converted into a basic score V1 of 0-100 through a linear mapping. Extract the top N features by weight in the model and calculate the extra points awarded for their values ​​deviating from the normal range. The score is fine-tuned based on the judgment results of the rule engine; The base score, bonus points, and fine-tuning scores are added together to obtain the final probability value P, which is in the range of 0-100.

[0012] Optionally, the dynamic optimization adjustment in step S4 includes: The initial threshold T is calculated and determined based on the distribution of P-values ​​between normal users and malicious crawlers in historical data; Based on the normal user misjudgment rate and malicious crawler missed judgment rate generated in actual operation, the threshold T is periodically adjusted according to the preset step size; When the system encounters a traffic peak, the threshold T is temporarily increased.

[0013] Optionally, the tiered defense strategy described in step S5 includes: When the P value falls within the slightly suspicious range, CAPTCHA verification is triggered. When the P-value falls within the moderately suspicious range, the access frequency is restricted; When the P-value falls within the highly suspicious range, temporary blocking measures will be implemented.

[0014] Optionally, the continuous optimization described in step S5 includes: Collect the judgment results from manual review, label the corresponding feature vectors, and add them to the model training set; Regularly update machine learning model parameters using incremental learning algorithms; Regularly generate analysis reports and dynamically adjust feature weights based on changes in crawler behavior.

[0015] The beneficial effects of this invention are that it provides a Java-based intelligent anti-malicious web crawler defense method, which solves the problems of poor protection effect, high false positive rate and weak adaptability of existing anti-web crawler methods, and achieves accurate identification and efficient interception of malicious web crawlers, while ensuring a good access experience for normal users. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the processing flow of an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the behavior analysis model structure according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in one or more embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] like Figures 1-2 As shown, one or more embodiments of this application illustrate a Java-based intelligent anti-malicious web crawler defense method, comprising: S1: Responsible for collecting multi-dimensional data during user access, including but not limited to: HTTP request header information (User-Agent, Referer, Cookie, etc.), access time series (request frequency, interval duration, active periods), behavioral trajectory data (page jump path, click location, dwell time), and network characteristics (IP address, port, geographical location, ISP information). S2: Efficient data processing based on Java's Steam streaming, generating feature vectors (which can be understood as a structured data set reflecting key characteristics of access behavior formed after processing the raw data in S1): S21. Perform structured transformation on the metadata in S1: For example, convert the messy text information in user access logs into tabular data containing fixed fields such as "access time, IP address, and requested URL". For instance, convert a log such as "2022-07-09 10:05:23 192.168.1.1 accessed the homepage" into a structured data entry containing three fields: time, IP address, and accessed page. S22. Use Z-Score (Z-Score is calculated as follows: for data x within a certain time range, its standardized value z = (x - μ) / σ, where μ is the average value of the data within that time range and σ is the standard deviation) to standardize numerical features: For numerical data such as "request frequency (times / minute)" and "page dwell time (seconds)," standardization eliminates the influence of different orders of magnitude. For example, if an IP user's request frequency is 60 times / minute, the calculated Z-Score is 1.2, meaning that this frequency is 1.2 standard deviations above the average level; S23. Perform one-hot encoding on categorical features: Convert non-numerical features such as "User-Agent type" and "access time period (morning / noon / evening)" into a numerical form that can be recognized by computers. For example, encode "User-Agent for Chrome browser" as [1,0,0], "User-Agent for Firefox browser" as [0,1,0], and "User-Agent for crawler feature" as [0,0,1]. S24. Extract temporal features of access behavior (e.g., number of requests within a sliding window): Using a 5-minute sliding window, count the number of user requests, page jumps, and other features that change over time within each window. For example, if a user made 8 requests between 9:00 and 9:05 and 12 requests between 9:05 and 9:10, these data will be included in the feature vector to reflect the temporal pattern of their access. S25. After the above processing, each user's access behavior will be transformed into a feature vector containing multi-dimensional data. For example, the feature vector of a certain access behavior might be: [Request frequency after Z-Score normalization 1.2, Chrome browser identifier after one-hot encoding [1,0,0], number of requests within a 5-minute sliding window 8, normalized page dwell time 0.8...]. This feature vector is like a "behavioral label" that can comprehensively and systematically reflect the key characteristics of the access behavior, providing a unified and effective analytical basis for rule engine matching and machine learning model classification in S3.

[0021] S3: Integrates rule engines and machine learning models to build a two-layer identification mechanism (which can be understood as accurately determining whether access behavior belongs to malicious crawlers through dual verification of "explicit rule screening + intelligent pattern recognition"): 1. Rule Engine: Like a pre-set "security checkpoint," the existing Drools rule engine pre-defines a series of clear and directly determinable rules to perform initial screening of access behavior. For example: 1.1) When a single IP address sends more than the system-defined threshold of x requests within one minute, the behavior is flagged as suspicious. Monitoring relies on the collaborative work of the data acquisition layer in S1 and the feature extraction module in S2. When a user initiates a request, the data acquisition layer intercepts it through the gateway's GlobalFilter component, capturing the source IP address and request timestamp of each request in real time and temporarily storing this information in a memory queue. Using Java's scheduled tasks, a statistical calculation is triggered every minute: the request records corresponding to the IP address in the queue are filtered and counted using a Stream to quickly calculate the total number of requests within one minute. Then, distributed caching tools such as Redis are used to associate and store the IP address with the corresponding request count, ensuring that in a distributed deployment environment, multiple server nodes can share the request count data for the same IP address, avoiding misjudgments due to incomplete statistics from a single node. 1.2) If the User-Agent field in the request header matches the system's built-in crawler signature database (e.g., containing keywords such as "bot" or "spider"), the IP request address will be identified as having crawler characteristics, marked, and stored in the database blacklist table. The blacklist table contains the following fields: IP address (varchar type, length 32, primary key, stores the tagged IP address, such as "192.168.1.1"); Triggering rule (varchar type, length 255, records the name of the triggered rule, which is fixed here as "User-Agent matching crawler feature library rule"); The time stamp (datetime type, records the specific time the stamp is made, such as "2025-07-11 10:00:00"); Modification time (datetime type, records the latest time of modification, such as "2025-07-11 10:00:00"); Status (tinyint type, length 1, 0 indicates the ban is active, 1 indicates the ban has been lifted); Unblocking time (datetime type; if it is a temporary ban, record the automatic unblocking time, such as "2025-07-11 11:00:00"; if it is a permanent ban, record null). Operator (varchar type, length 64, records the account of the person who performed the last operation; "system" is used when the system performs automatic operations); 1.3) When an IP address repeatedly accesses multiple non-existent URLs within a short period, resulting in frequent 404 errors (e.g., 10 or more 404 errors within 10 seconds), this will be identified as abnormal behavior and recorded in the Abnormal Behavior Alert Table. The structure of the Abnormal Behavior Alert Table is as follows: Alarm ID (bigint type, auto-incrementing primary key, uniquely identifies each alarm record); IP address (varchar type, length 32, stores the IP address that triggered the abnormal behavior, such as "192.168.1.2"); Error start time (datetime type, records the time when the first 404 error occurred, such as "2025-07-11 10:05:00"); Abnormal end time (datetime type, records the time when the abnormal behavior stopped, such as "2025-07-11 10:05:10"); 404 error count (int type, records the total number of 404 errors generated during the abnormal period); A list of invalid URLs accessed (text type, storing non-existent URL paths accessed by this IP, multiple URLs separated by commas); Alarm status (tinyint type, length 1, 0 indicates unprocessed, 1 indicates processed, 2 indicates false alarm); Processing time (datetime type, records the time when the alarm processing was completed; null if not processed); Processor (varchar type, length 16, records the name of the person who handled the exception information); Processing notes (varchar type, length 512, records the notes for processing this alarm, such as "User has been notified to check access path").

[0022] 1.4) Use Java automated tasks to periodically scan the blacklist and abnormal behavior tables above, and push the information (via email, WeChat, SMS, etc.) to the relevant responsible persons. 2. Machine Model Construction and Training: A classification model M is built using Java machine learning libraries (such as Weka and Deeplearning4j). Model M learns the feature vector of each historical access data in S2 and outputs the probability value P of each access record belonging to a "malicious crawler". For example, after analyzing the feature vector of a certain behavior (request frequency normalization value 1.8, crawler User-Agent encoding [0,0,1], 5-minute window request count 22), the model outputs P=0.85 (i.e., 85% probability of being a malicious crawler). The specific steps are as follows: 2.1) Set the “fuzzy range” of P (e.g., 0.3~0.7). If P≥0.7 or P≤0.3, proceed directly to V value calculation; if P is in the fuzzy range, trigger the S3 rule engine for secondary verification (e.g., check whether the IP is in the blacklist or whether it hits the high-frequency request rule), and use the rule result as the basis for P correction (e.g., if the rule is judged to be suspicious, P is increased by 0.2).

[0023] 2.2) Convert P to a base score of 0-100 using a linear mapping: V1 = P × 100 (rounded to the nearest integer). For example, P = 0.85 → V1 = 85.

[0024] 2.3) Extract the top 5 features by weight in the model (such as "number of requests in 5 minutes", "User-Agent type", and "standardized dwell time"), and award points to features that deviate from the normal range. When a single feature deviates from the normal user mean by n standard deviations and the feature weight is w, the extra score is n×w×5 (5 is a fixed coefficient, and the total score is ≤100). Example: If the number of requests in 5 minutes deviates from the normal mean by 3 standard deviations (weight 0.3), then the extra points = 3 × 0.3 × 5 = 4.5 → P = 85 + 4 = 89 (rounded).

[0025] 2.4) Fine-tune P based on the judgment results of the S3 rule engine. If the "User-Agent matching crawler feature library" rule is triggered, P will increase by 10 points. If the "requests exceed threshold within 1 minute" rule is triggered, P will increase by 5 points; If the "frequent 404 errors" rule is triggered, P will increase by 8 points; The corrected P value still remains in the range of 0-100 (e.g., P=95+10→P=100).

[0026] S4: Threshold setting and dynamic optimization.

[0027] 1. Analyze the distribution of P-values ​​in historical data. The 95th percentile of the P-value for normal users is T1 (e.g., 40), and the 5th percentile of the P-value for malicious crawlers is T2 (e.g., 60). The initial threshold is T = (T1 + T2) / 2 (e.g., 50). 2. Distributed dynamic threshold adjustment Multi-node threshold synchronization is implemented based on ZooKeeper, and adjustments are made daily according to the following logic: If the false positive rate of normal users (P≥T but actually normal) is >3%, T is increased by 5 (e.g., 50→55). If the false positive rate of malicious crawlers (P < T but it is actually a crawler) is greater than 5%, T is reduced by 5 (e.g., 50 → 45). During peak traffic periods (such as major e-commerce promotions), the threshold is automatically relaxed (T is temporarily increased by 10) to prevent normal users from being mistakenly blocked.

[0028] S5: Final decision and continuous optimization.

[0029] 1. Grading determination based on P-value Compare P with the current threshold T, and output the judgment result for tiered defense: Slightly suspicious: P∈[T×0.8,T) → trigger CAPTCHA verification; Moderately suspicious: P∈[T,T×1.5) → Limit access frequency; Highly suspicious: P≥T×1.5 → Temporarily ban IP / account.

[0030] 2. Feedback and optimization mechanism Collect the manual review results from S3 (e.g., P=60 is judged as a misjudgment), label the corresponding feature vectors and add them to the training set; The model parameters are updated weekly using an incremental learning algorithm, with a focus on optimizing the P-value output accuracy in the fuzzy range (0.3~0.7). A monthly P-value distribution report (based on JFreeChart) is generated to analyze the impact of crawler behavior on the P-value and adjust feature weights (e.g., increase the weight of a feature when the crawler begins to disguise its dwell time).

[0031] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0033] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0034] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0035] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0036] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A Java-based intelligent anti-malicious web crawler defense method, characterized in that, Includes the following steps: S1: Collect multi-dimensional metadata of user access requests; S2: Perform structured processing and feature engineering on the multi-dimensional metadata to generate feature vectors for behavior analysis; S3: Construct and apply a two-layer identification mechanism that integrates a rule engine and a machine learning model, and calculate the probability value P that the user access behavior belongs to a malicious crawler based on the feature vector; S4: Set an initial judgment threshold T based on historical data statistical analysis, and dynamically optimize and adjust the judgment threshold T based on the false judgment rate and the false negative rate; S5: Compare the probability value P with the current judgment threshold T, execute the hierarchical defense strategy based on the comparison result, and continuously optimize the machine learning model and feature weights based on the feedback of the defense result.

2. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, In step S1, the multi-dimensional metadata includes: HTTP request header information, access time series data, user behavior trajectory data, and network feature data.

3. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, Step S2 includes: S21: Perform a structured transformation on the metadata to generate structured data containing fixed fields; S22: Perform Z-Score standardization on numerical features to eliminate the influence of different orders of magnitude; S23: Perform one-hot encoding on categorical features; S24: Extract the temporal features of access behavior; S25: Finally, the data processed by steps S21 to S24 are combined into a feature vector representing user access behavior.

4. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, The rule engine described in step S3 has at least one of the following preset rules: Rule 1: Monitor the number of requests made by a single IP address within a unit of time. If the number exceeds a preset threshold, it will be marked as suspicious. Rule 2: Check the User-Agent field in the request header. If it matches the built-in crawler feature library, it is determined to have crawler features. Rule 3: Monitor the frequency of IP addresses triggering 404 errors within a short period of time. If the frequency exceeds a preset threshold, it will be identified as abnormal behavior.

5. The Java-based intelligent anti-malicious web crawler defense method according to claim 4, characterized in that, The rules engine's operations include: recording the IP address information that triggers the rule to the blacklist or abnormal behavior alarm table, and periodically pushing the alarm information to the person in charge.

6. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, The machine learning model described in step S3 is a classification model, which outputs a probability value P through the following steps: The model is trained using feature vectors from historical access data; Input the real-time feature vector into the trained model to obtain the basic probability value P that the access behavior belongs to a malicious crawler. If the P value is within a preset fuzzy range, the rule engine is triggered to perform a second verification, and the P value is corrected based on the verification result.

7. The Java-based intelligent anti-malicious web crawler defense method according to claim 6, characterized in that, After obtaining the base probability value P, the final probability value P is calculated further through the following steps: The basic probability value P is converted into a basic score V1 of 0-100 through a linear mapping. Extract the top N features by weight in the model and calculate the extra points awarded for their values ​​deviating from the normal range. The score is fine-tuned based on the judgment results of the rule engine; The base score, bonus points, and fine-tuning scores are added together to obtain the final probability value P, which is in the range of 0-100.

8. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, The dynamic optimization adjustment mentioned in step S4 includes: The initial threshold T is calculated and determined based on the distribution of P-values ​​between normal users and malicious crawlers in historical data; Based on the normal user misjudgment rate and malicious crawler missed judgment rate generated in actual operation, the threshold T is periodically adjusted according to the preset step size; When the system encounters a traffic peak, the threshold T is temporarily increased.

9. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, The tiered defense strategy described in step S5 includes: When the P value falls within the slightly suspicious range, CAPTCHA verification is triggered. When the P-value falls within the moderately suspicious range, the access frequency is restricted; When the P-value falls within the highly suspicious range, temporary blocking measures will be implemented.

10. The Java-based intelligent anti-malicious web crawler defense method according to claim 1, characterized in that, The continuous optimization described in step S5 includes: Collect the judgment results from manual review, label the corresponding feature vectors, and add them to the model training set; Regularly update machine learning model parameters using incremental learning algorithms; Regularly generate analysis reports and dynamically adjust feature weights based on changes in crawler behavior.

Images