Anti-webpage crawler method based on user access time portrait

Abstract

An anti-webpage crawler method based on a user access time portrait comprises the steps that firstly, a personalized reading time reference model is constructed by extracting webpage content features to replace a traditional static rule, and a foundation is laid for accurate judgment; secondly, a multi-level progressive defense mechanism of'suspected crawler users-highly suspected targets-multi-level security challenges' is adopted, and it is ensured that non-interference passing is achieved for most of normal users while it is intercepted that the crawlers are intercepted; and finally, establishing a refined access control system comprising a user white list, a black list and a crawler white list, and continuously driving self-learning and calibration of the model by using white list user data, so that the system has long-acting self-adaptive protection capability.

Description

Technical Field

[0001] This invention belongs to the field of information security technology, and relates to web crawler identification and protection technology, specifically to an anti-web crawler method based on user access time profiles. Background Technology

[0002] With the rapid development of internet information technology and the increasing prominence of data value, web crawlers have triggered a series of serious security problems, such as large-scale data theft, excessive website server load, leakage of core business assets, and unauthorized collection of personal privacy data. Therefore, anti-web crawler technology has become an indispensable and crucial protective measure for all types of websites, especially information, knowledge-based payment, and data service platforms whose core competitiveness lies in content.

[0003] Currently, mainstream anti-web crawler solutions exhibit significant technical limitations and adaptability bottlenecks when facing increasingly sophisticated web crawler technologies. Their fundamental flaws lie in the severe disconnect between protection logic and webpage semantics, as well as the lack of dynamic evolution capabilities in their overall mechanisms.

[0004] Specifically, most traditional solutions rely on static rules (such as fixed access frequency thresholds) for a "one-size-fits-all" judgment. This mechanism not only suffers from a high false positive rate for normal users because it cannot distinguish the reasonable time difference required to access short pages and in-depth long articles, but its rigid rules are also ineffective against "slow crawlers" that can deliberately reduce their request frequency. Especially with the development of AI technology, this limitation has been amplified dramatically: AI crawlers with learning capabilities can dynamically simulate human browsing rhythms, causing their access patterns to fall entirely within the "normal" range defined by static thresholds, thus easily evading detection.

[0005] Secondly, after passively identifying suspected behavior, traditional solutions generally employ interrupt-type CAPTCHAs for challenge. While this method can block some low-level web crawlers, its crude interaction severely damages the user experience. More importantly, when facing AI web crawlers with integrated image recognition and cracking capabilities, the protection provided by such simple CAPTCHAs is gradually failing, rendering this defense virtually ineffective.

[0006] In summary, traditional solutions fail to establish an intrinsic link between "access behavior" and "accessed content" in terms of protection logic, and lag behind the anthropomorphic and intelligent evolution of AI web crawlers in terms of technical means. Therefore, there is an urgent need in this field for a next-generation intelligent anti-web crawler method that can deeply understand web page content, accurately distinguish between humans and various types of web crawlers (especially advanced AI crawlers), and balance robust protection with a smooth user experience. Summary of the Invention

[0007] The main objective of this invention is to design an anti-web crawler method based on user access time profiles. By extracting webpage content features, a personalized reading time benchmark model is established to accurately distinguish between human users and crawler programs. While reducing the false positive rate, a progressive security challenge mechanism is adopted to optimize the user experience, and the model is continuously calibrated through whitelisted user data to achieve adaptive intelligent protection.

[0008] An anti-web crawling method based on user access time profiling includes the following steps:

[0009] Step 1, Webpage Content Analysis: Use big data models to analyze webpage content and extract content feature tags that affect reading time, including volume tags, content type tags, and complexity tags;

[0010] Step 2, Reading Time Benchmark Model Construction: Based on content feature tags and their corresponding historical user reading time data, a machine learning regression algorithm is used for fitting and training to establish a mapping relationship from content feature tags to reading time intervals, and to calculate the normal reading time intervals for users;

[0011] Step 3, Reading Time Monitoring and Anomaly Identification: Monitor the reading time of users on the target webpage, calculate the effective reading time based on client behavior data, and compare the effective reading time with the normal reading time range; if the reading time is not within the range, mark the user as a suspected web crawler; if the reading time is within the range, mark the user as a whitelisted user.

[0012] Step 4, Multi-layered security challenges: If a suspected web crawler is marked as a suspected web crawler for the second time within a preset time window, they will be listed as a highly suspicious target and a multi-layered security challenge will be triggered; the challenge includes seamless behavior verification, device fingerprint verification, and interactive challenge;

[0013] Step 5, List Management and Access Control: Users who fail the security challenge will be blacklisted and their subsequent access will be restricted; users who pass the challenge will be whitelisted and exempted from anomaly detection and security challenges for a short period of time; the whitelist has an update cycle, and permissions will be reset after expiration; in addition, a separate crawler whitelist is set up to store the identification information of known benign web crawlers.

[0014] Step 6, continuous model optimization: aggregate and analyze the reading time data of whitelisted users, and store it in association with web page content feature tags; when the amount of data accumulates to a preset threshold, trigger the update process of the reading time benchmark model, and recalculate and calibrate the model using the newly added data.

[0015] The beneficial effects achieved by this invention are as follows: (1) In terms of accuracy, through personalized analysis of content association, it effectively distinguishes between humans and crawlers, especially improving the ability to identify advanced slow crawlers; (2) In terms of user experience, through non-intrusive monitoring and progressive challenge strategies, it perfectly balances security protection and access smoothness; (3) In terms of system vitality, through a closed-loop adaptive learning mechanism, it can continuously evolve and cope with the ever-changing crawling methods in the future. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of the anti-web crawler method in a specific embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] First, this embodiment provides the following definitions:

[0019] Reading Time Benchmark Model: Based on webpage content feature tags and historical user reading time data, this model is trained using a machine learning regression algorithm to predict the time interval required for a normal user to read webpages and paragraphs.

[0020] Content feature tags: Features extracted from web page content to quantify factors affecting reading time, including volume tags (quantifying the amount of web page content), content type tags (identifying the main content type of the web page), and complexity tags (characterizing the difficulty of reading and understanding the web page content).

[0021] Suspected web crawler users: Users whose effective reading time is outside the normal reading time range are marked by the system as users suspected of web crawling.

[0022] Multi-layered security challenges: A series of security verification measures for highly suspicious targets, including seamless behavioral verification, interactive challenges, and device fingerprint verification, to ultimately confirm user identity.

[0023] Whitelist: Stores the identification information of users who have passed security challenges or are known to be trusted. Users on the list are exempt from anomaly detection and security challenges.

[0024] Blacklist: Stores identification information of users identified as web crawlers or malicious users. Access will be blocked for users on the list.

[0025] This invention designs an anti-web crawler method based on user access time profiles, mainly including the following steps:

[0026] Step 1, Webpage Content Analysis: Use big data models to analyze webpage content and extract multiple content feature tags that affect reading time, including volume tags, content type tags, and complexity tags.

[0027] Step 2, Reading Time Benchmark Model Construction: Based on the content feature tags of a large number of web page samples and their corresponding historical user reading time data, a machine learning regression algorithm (such as gradient boosting decision tree, random forest, etc.) is used for fitting and training to establish a mapping relationship from content feature tags to reading time intervals, and the normal user reading time interval is calculated.

[0028] Step 3, Reading Time Monitoring and Anomaly Identification: Monitor the effective reading time of users on the target webpage. Calculate the effective reading time based on client behavior data (such as page load completion timestamp, page leave event trigger timestamp, and page visibility status change events), and compare the effective reading time with the normal reading time range. If the reading time is outside the range, the user is marked as a suspected web crawler.

[0029] Step 4, Multi-layered Security Challenges: If a suspected web crawler is flagged a second time within a preset time window, they are classified as a highly suspicious target, triggering multi-layered security challenges. These challenges include seamless behavior verification (analyzing mouse movement trajectories, scrolling rhythm, etc.), device fingerprint verification, and interactive challenges (CAPTCHA tasks). Challenges are executed progressively according to a strategy, with seamless verification taking priority.

[0030] Step 5, List Management and Access Control: Users who fail the security challenge will be blacklisted, and their subsequent access will be restricted; users who pass the challenge will be whitelisted, exempting them from anomaly detection and security challenges for a short period. The whitelist has an update cycle, and permissions are reset upon expiration. In addition, a separate web crawler whitelist is established to store the identification information of known benign web crawlers.

[0031] Step 6, Continuous Model Optimization: Aggregate and analyze the reading time data of whitelisted users, and store it in association with webpage content feature tags. When the data volume accumulates to a preset threshold, the reading time benchmark model update process is triggered, and the model is recalculated and calibrated using the newly added data.

[0032] This method will be implemented using a large news and information website as an example.

[0033] Step 1: Webpage content analysis and tag extraction.

[0034] S101: Target webpage selection: Select a webpage that contains typical in-depth reporting articles as the analysis target.

[0035] S102: Content Feature Tag Extraction: Using big data analysis models to parse webpage content and extract standardized content feature tag vectors.

[0036] S1021: Extract volume tag X: Extract multimedia elements from the webpage through structured parsing and construct vector X=[3850,6,1], which correspond to: number of text characters (3850 characters), number of accompanying images (6 images), and number of embedded videos (1 video).

[0037] S1022: Extract content type tag Y:

[0038] Referring to the resource classification logic of the national standard GB / T 20093.1-2022 and combining it with the characteristics of news business, content types are mapped to standard numerical tags. In this case, the target webpage is identified as "in-depth report" by the NLP classification model, so Y=05.

[0039] S1023: Extracting the complexity label Z:

[0040] Referring to the Chinese Readability Assessment model, a complexity calculation model was obtained through multiple linear regression fitting. The reading difficulty coefficient Z (range 0.0-1.0) was calculated. The calculation formula is defined as follows: Z = 0.04R1 (logical paragraph score) + 0.06R2 (average sentence length) - 0.01R3 (technical terminology density).

[0041] in:

[0042] The logical paragraph score R1 is used to measure the degree of structure in an article, and the calculation formula is as follows: Parameter N 段落 R1 represents the number of paragraphs. If the number of paragraphs is greater than or equal to 10, then R1 = 10.

[0043] The average sentence length R² reflects the complexity of the sentence structure. .

[0044] The R3 density of technical terms is calculated by comparing it with industry-specific thesaurus using word segmentation tools (such as Jieba or LTP). .

[0045] Calculation Example: If the current webpage has 23 logical paragraphs (R1=3), an average sentence length of 40 characters (R2=40), and a term density of 5% (R3=4), substitute the following into the formula:

[0046]

[0047] Note: In practical applications, the final Z value is usually normalized to the [0,1] interval using Min-Max. In this example, the calculated result of 0.75 represents the standard difficulty after normalization.

[0048] Feature integration: The final generated standardized feature label vector is: [(3850,6,1),05,0.75]. This vector will serve as the standard input to the machine learning model in steps S201 and S202.

[0049] Step 2, constructing the reading time benchmark model.

[0050] S201: Preparation and integration of model training data and model construction.

[0051] S2011: Multidimensional Data Acquisition and Preprocessing: Approximately 1.2 million webpage samples covering different vertical fields were extracted from anonymized business logs of multiple collaborative content websites. Using the feature extraction algorithm from step 1, each webpage was mapped to a standardized content feature tag vector V=[X,Y,Z].

[0052] S2012: User Behavior Tagging: Accurately calculate the effective reading time T of a large number of anonymous users by synchronizing front-end tracking and back-end logs. effective The calculation formula is: T effective =( T leave -T load ). T leave This indicates the time from when a user enters the page to when they leave it. T load This indicates page load time. To eliminate noise, a data cleaning strategy is implemented to remove abnormal samples with a dwell time of <3 seconds (accidental touch) or >1 hour (idle state), ensuring that the tag data can accurately reflect user engagement.

[0053] S2013: Training Data Pair Construction: Integrate data from multiple source sites to construct a training sample set. , where {T} is the set of historical user reading time distributions for the corresponding webpage. This cross-site integration method ensures sufficient diversity in sample size, type, and complexity.

[0054] S2014: Construction of a prediction model based on DeepSeek-14B:

[0055] (1) Basic model selection: DeepSeek-R1-Distill-Qwen-14B was adopted as the underlying pre-trained model. This model has powerful semantic understanding and context modeling capabilities, and can deeply capture the non-linear relationship between web page content features and reading behavior.

[0056] (2) Adapter Fine-tuning: A regression task adaptation layer is added to DeepSeek-14B, and the above 1.2 million labeled samples are used for reinforcement training. By introducing negative sampling and contrastive learning, the model learns how to predict the probability density distribution of the target reading time based on "content volume, type, and complexity".

[0057] (3) Enhanced generalization: Thanks to the general knowledge accumulated by the DeepSeek model on massive corpora, combined with the multi-site-specific features integrated in this step, the trained model can not only accurately predict the reading time of known types of web pages, but also has a strong generalization prediction ability when dealing with novel or long-tail content.

[0058] S202: Training and Output of Machine Learning Models

[0059] S2021: Constructing a High-Standard Evaluation Benchmark (Test Set): To verify the model's reliability in real-world scenarios, a separate high-quality test set was constructed in addition to the training set integrated above. This set was created by accessing the open-source Sogou News Corpus or the THUCNews Chinese Text Classification Dataset, combined with "Golden Datasets" manually annotated by senior editors during the business internal testing phase. The test set covers core channels such as politics, technology, and finance, ensuring the evaluation benchmark's authority and objectivity, and is used to ultimately measure the consistency between predicted values ​​and the median actual reading time.

[0060] S2022: Define the regression objective: Use the median of historical reading time data as the label value for supervised learning. Compared to the mean, the median is more robust to abnormal fluctuations in the long-tail distribution (such as excessively long dwell times caused by users leaving the computer idle), and can more realistically reflect the "baseline reading time" of web page content.

[0061] S2023: LightGBM Supervised Learning Training Process: Utilizing the efficient histogram-based algorithm of the LightGBM framework, the non-linear mapping from multi-dimensional features to baseline time is achieved through the following specific steps:

[0062] (1) Feature Engineering and Encoding: The [X, Y, Z] vector in S201 is preprocessed. Categorical features (such as content type) are categorically encoded, and numerical features are normalized and used as model input.

[0063] (2) Parameter configuration initialization: Set the objective function to the regression task (objective='regression') and the evaluation metric to the mean absolute error (metric='mae'). Configure hyperparameters such as num_leaves (number of leaf nodes) and learning_rate (learning rate) to balance model depth and generalization.

[0064] (3) Iterative optimization process:

[0065] Weak learner construction: The model uses a leaf-wise growth strategy to generate decision trees. In each iteration, LightGBM calculates the gradient between the current predicted value and the true median, and fits a new tree to compensate for the residuals.

[0066] Feature parallelization: Utilizing its efficient feature parallelization capabilities, it quickly identifies the weight of the impact of "content volume" and "complexity" on reading time under specific combinations.

[0067] (4) Training stopping strategy: An early stopping mechanism is introduced. If the MAE performance of the model on the above news test set no longer improves in 50 consecutive iterations, training is stopped to prevent overfitting.

[0068] S2024: Model Output and Performance Evaluation: Optimal Nonlinear Mapping Function After Training It can output the expected reading time of a webpage in real time. Experimental results show that the model's mean absolute error (MAE) on a high-quality news test set is controlled within 15%. This means that for a webpage with a baseline reading time of 100 seconds, the model's prediction error averages no more than 15 seconds, and its accuracy is sufficient to support subsequent business operations for the graded evaluation of user engagement and the optimization of recommendation algorithms.

[0069] S203: Determine the normal user reading time interval: For the target webpage in the implementation phase (i.e., the in-depth report article in step 1), after extracting its content feature tag vector [(3850,6,1),05,0.75], it is input into the pre-trained reading time benchmark model. The model outputs a benchmark reading time median T = 85 seconds. Considering the differences in individual user reading speeds, the normal user reading time interval is not a fixed value, but is given as a confidence interval around T. This interval is determined by calculating the 5th percentile (P5) and 95th percentile (P95) of the reading time of users on similar webpages in the training data, for example, [P5, P95] = [55 seconds, 135 seconds]. This interval is the personalized and dynamic benchmark for judging whether the accessing user behavior is abnormal in subsequent steps, and can cover about 90% of normal user behavior.

[0070] Step 3, Access Monitoring and Anomaly Identification.

[0071] S301: Pre-list screening: User A's device fingerprint is not in the whitelist or blacklist when accessing the site for the first time, and the system enters the evaluation process.

[0072] S302: Calculation of Effective Reading Time: Based on client behavior data (page load completion timestamp, page leave timestamp, page visibility status change event), the system accurately calculates that user A's effective reading time on this article page is 32 seconds. The calculation process is: (page leave timestamp - page load completion timestamp) - total page invisibility time = 32 seconds.

[0073] S303: Abnormal Behavior Detection: User A's actual reading time (32 seconds) is significantly lower than the lower limit of the normal range (55 seconds). The system marks them as a "suspected web crawler user" and records this abnormal event and timestamp. If the reading time is within the normal range, they will be directly added to the whitelist.

[0074] Step 4: Multi-layered security challenges and solutions.

[0075] S401: Risk Escalation: If User A accesses another in-depth analysis article again within a preset time window of 1 hour, the reading time (28 seconds) will also be judged as abnormal, and the second time will be marked as "suspected web crawler user".

[0076] S402: Triggering a multi-layered security challenge: For users who are repeatedly marked as "suspected web crawlers", the system raises their risk level to "highly suspected target" and initiates a progressive challenge.

[0077] S4021: Phase One (Seamless Verification):

[0078] (1) Behavioral data collection and mechanized pattern recognition:

[0079] The system utilizes a front-end JavaScript event listening mechanism to collect user behavior characteristics in parallel.

[0080] Data collection steps: Mousemove is used to monitor mouse coordinates (x, y) and timestamp (t), while wheel or touchmove is used to monitor page scrolling frequency, pixel displacement, and pause intervals. After high-frequency sampling, the data is compressed using the Douglas-Peucker algorithm and transmitted asynchronously to the backend.

[0081] Behavioral analysis: The system found through analysis that user A's trajectory presents a mechanized grid pattern (characterized by a path curvature of almost zero, uniform linear movement, and a lack of subtle tremors and acceleration changes characteristic of human operation), and completely lacks the "scroll-pause-read" rhythm that matches the content complexity (Z value defined in step S102).

[0082] (2) Device fingerprint association verification:

[0083] Device fingerprints are unique cryptographic identifiers generated by collecting the underlying characteristics of terminal devices.

[0084] Data collection dimensions: The system extracts data including Canvas drawing and rendering features, browser plugin list, system font fingerprint, GPU accelerator model, CPU core count, and real network parameters exposed by WebRTC.

[0085] Risk verification: The collected fingerprint information is immediately searched in the blacklist database. Analysis revealed that User A's device fingerprint has a high probability of association with a batch of previously confirmed distributed crawler clusters, with a feature overlap rate exceeding 95%.

[0086] (3) Comprehensive risk scoring and dynamic threshold determination

[0087] The system calculates the comprehensive risk score in real time, which uses a weighted scoring model.

[0088]

[0089] Among them W track For trajectory weights, W device For device fingerprint weights, W network S represents the network weight (e.g., trajectory weight 0.4, device fingerprint weight 0.4, network weight 0.3), and S represents the anomaly score for each individual item.

[0090] First preset threshold: refers to the critical score at which "untrusted access" is determined and interception or secondary verification is triggered.

[0091] Initial settings and adjustments: In the current production environment in 2026, the initial threshold reference ROC curve is set to the optimal equilibrium point of 65 points.

[0092] Adaptive optimization: This threshold has a dynamic closed-loop adjustment mechanism. If the feedback from the subsequent second-stage verification shows an increase in the false positive rate, the system will automatically raise the threshold through Bayesian inference; if it is a sensitive period (such as when a large-scale credential stuffing attack occurs), the risk control engine will automatically lower the threshold to tighten access control.

[0093] Conclusion: Since User A's comprehensive risk score has exceeded the first preset threshold, the system immediately triggers the second-stage secondary verification logic.

[0094] S4022 Phase 2 (Interactive Challenge):

[0095] On user A's next request, the system triggers the selection of a verification code for final confirmation.

[0096] S403: Challenge Result Handling: User A failed the CAPTCHA challenge. The system officially blacklists User A and blocks further access. The returned rejection page provides a link to an online appeal form. If User A had passed the CAPTCHA challenge, User A would have been added to the whitelist.

[0097] For example, suppose user C is also triggered with a challenge, but passes the seamless verification (natural behavior pattern), then there will be no further interference, and access will be directly allowed.

[0098] Step 5: List Management and Model Optimization.

[0099] S501: Whitelist Management: Add users who pass security challenges or are known trusted users (such as logged-in users or partner IPs) to the whitelist, exempting them from detection for a period of time (e.g., 7 days).

[0100] S502: Crawler Whitelist: For crawlers that have been registered with search engines (such as Googlebot), add their identification information to a separate crawler whitelist to completely exempt them from the detection and challenges in the aforementioned steps.

[0101] S503: Web Crawler Blacklist: Users who fail the detection will be blacklisted. The blacklist will be updated every 30 days and will record the fingerprints of users who have been blacklisted. Users who are blacklisted a second time will not be removed from the blacklist when it is updated. If a user wants to be removed from the blacklist early, they must submit relevant proof to the administrator via the website email address.

[0102] Step 6, continuous model optimization:

[0103] The system continuously collects the effective reading time of the aforementioned in-depth report articles from whitelisted users. When the cumulative effective reading time data samples collected for this article exceed 500 within 24 hours, the model update process is triggered. Using this new batch of high-reliability data, the system employs an incremental learning mechanism for model fine-tuning: low-rank adaptation of the DeepSeek-14B attention weights is performed using LoRA technology, and the warm start function of LightGBM is used to fit the residuals of the new samples onto the existing tree structure. This process strictly controls the learning rate to avoid catastrophic forgetting, ensuring that the model steadily improves its global generalization ability while absorbing the latest reading behavior features. After the update, the baseline reading time range for this article is optimized to [50 seconds, 130 seconds], thus more accurately reflecting the reading habits of real users and achieving the system's self-evolution and calibration.

[0104] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A method for preventing web crawling based on user access time profiles, characterized in that: The method includes the following steps: Step 1, Webpage Content Analysis: Use big data models to analyze webpage content and extract content feature tags that affect reading time, including volume tags, content type tags, and complexity tags; Step 2, Reading Time Benchmark Model Construction: Based on content feature tags and their corresponding historical user reading time data, a machine learning regression algorithm is used for fitting and training to establish a mapping relationship from content feature tags to reading time intervals, and to calculate the normal reading time intervals for users; Step 3, Reading Time Monitoring and Anomaly Identification: Monitor the reading time of users on the target webpage, calculate the effective reading time based on client behavior data, and compare the effective reading time with the normal reading time range; If the reading time is outside the specified range, the user will be marked as a suspected web crawler. If the reading time is within the specified range, the user will be marked as a whitelisted user. Step 4, Multi-layered security challenges: If a suspected web crawler is marked as a suspected web crawler for the second time within a preset time window, they will be listed as a highly suspicious target and a multi-layered security challenge will be triggered; the challenge includes seamless behavior verification, device fingerprint verification, and interactive challenge; Step 5, List Management and Access Control: Users who fail the security challenge will be blacklisted and their subsequent access will be restricted; users who pass the challenge will be whitelisted and exempted from anomaly detection and security challenges for a short period of time; the whitelist has an update cycle, and permissions will be reset after expiration; in addition, a separate crawler whitelist is set up to store the identification information of known benign web crawlers. Step 6, continuous model optimization: aggregate and analyze the reading time data of whitelisted users, and store it in association with web page content feature tags; when the amount of data accumulates to a preset threshold, trigger the update process of the reading time benchmark model, and recalculate and calibrate the model using the newly added data.

2. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: Step 1 includes the following sub-steps: S101: Target Webpage Selection: Select a webpage as the analysis target; S102: Content Feature Tag Extraction: Use big data analysis models to parse webpage content and extract standardized content feature tag vectors: Volume tag X, including text word count, number of images, and number of embedded videos; Content type tag Y, which provides the identification code of webpage content through an NLP classification model; Complexity tag Z, which calculates the reading difficulty coefficient based on features such as average sentence length, density of technical terms, and number of logical paragraphs.

3. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: Step 2 includes the following sub-steps: S201: Preparation and Integration of Model Training Data: Obtain a large-scale and diverse set of webpage samples from anonymized business logs of multiple collaborative content websites; for each webpage sample, execute the method in step 1 to extract its standardized content feature tag vector, i.e., [volume tag value, content type tag value, complexity tag value]; simultaneously, collect the effective reading time data of a large number of anonymous real users on these webpages through front-end tracking and back-end logs, i.e., (page exit timestamp - page load completion timestamp) - total page invisibility time, and filter out obviously abnormal records; the format of a training data sample is represented as {content feature tag vector, corresponding historical access user reading time data set}; S202: Training and Output of the Machine Learning Model: The LightGBM regression algorithm, a gradient boosting decision tree algorithm in machine learning, is used. The content feature tag vectors of the web pages in the aggregated dataset are used as input features, and the median historical reading time of a large number of users is used as the prediction target. Supervised learning training is performed on the model. The learning objective of the model is to find the optimal nonlinear mapping function F(X, Y, Z) → T from the multidimensional content features to the baseline reading time T. The model is evaluated on the test set, and the mean absolute error (MAE) between its predicted values ​​and the actual median reading time is controlled within 15%, which meets the application requirements. S203: Determine the normal user reading time interval: For the target webpage in the implementation phase, after extracting its content feature tag vector, it is input into the pre-trained reading time benchmark model. The model outputs the benchmark reading time median T. The 5th percentile and 95th percentile of the reading time of users of similar webpages in the training data are calculated to determine the benchmark reading time interval of the webpage.

4. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: Step 3 includes the following sub-steps: S301: Pre-list screening: User A's device fingerprint is not in the whitelist or blacklist when accessing the site for the first time, and the process is put into evaluation. S302: Calculation of effective reading time: Based on client behavior data, including page load completion timestamp, page leave timestamp, and page visibility status change events, the system calculates the effective reading time of user A on the article page. The calculation process is (page leave timestamp - page load completion timestamp) - total page invisibility time. S303: Abnormal behavior determination: User A's actual reading time is lower than the lower limit or higher than the upper limit of the reading time benchmark interval; The system flagged the user as a suspected web crawler and recorded the anomaly and its timestamp.

5. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: Step 4 includes the following sub-steps: S401: Risk Escalation: If User A visits another webpage again within a preset time window, their reading time will also be judged as abnormal, and they will be marked as a suspected web crawler user the second time. S402: Triggering a multi-layered security challenge: The system raises the risk level of the target to highly suspicious and initiates a progressive challenge; The first stage is contactless verification: the system performs contactless behavior verification and device fingerprint verification in parallel; it analyzes user A's mouse movement trajectory and natural scrolling behavior, and determines whether the device fingerprint is associated with a known web crawler. The second phase is an interactive challenge: on user A's next request, the system triggers a CAPTCHA selection for final confirmation. S403: Challenge Result Handling: If User A fails the CAPTCHA challenge, the system will officially blacklist him / her and block his / her subsequent access. The returned rejection page will provide a link to an online appeal form. If he / she passes the CAPTCHA challenge, User A will be added to the whitelist.

6. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: Step 5 includes the following sub-steps: S501: Whitelist Management: Add trusted users who pass security challenges or are known to be trusted to the whitelist, exempting them from detection during the validity period; S502: Crawler Whitelist: For crawlers that have been registered with search engines, add their identification information to a separate crawler whitelist, completely exempting them from the detection and challenges in the aforementioned steps.

7. The anti-web crawler method based on user access time profile according to claim 1, characterized in that: In step 6, the system continuously collects the effective reading time of whitelisted users on web pages. When the accumulated effective reading time data samples collected for web pages reach the threshold, the model update process is triggered. Using this new batch of high-reliability data, the original model was fine-tuned using incremental learning. After the update, the reading time benchmark range of the webpage was optimized, enabling the system to self-evolve and calibrate.

Images