A method, apparatus, and storage device for keeping a smart website alive.

Abstract

This invention relates to an intelligent website keep-alive method, apparatus, and storage device, belonging to the field of cloud protection technology. By performing minute-level statistical analysis on real-time website request data and combining it with a website model to derive the maximum request volume under multiple thresholds, a security load upper limit benchmark value and a load risk coefficient are calculated. Based on the coefficient value range, tiered keep-alive measures are implemented. After summarizing relevant data through a large model, intelligent alerts are sent to the cloud protection operation end and the user end via an alarm channel. This invention achieves accurate pre-emptive prediction and tiered intelligent handling of website load risks, eliminating the need for unlimited website expansion, reducing construction and maintenance costs, ensuring continuous and normal website service, and exhibiting strong adaptability and ease of deployment.

Description

Technical Field

[0001] This invention relates to the field of cloud protection technology, and in particular to a method, apparatus and storage device for keeping a smart website alive. Background Technology

[0002] User websites open to the internet are constantly bombarded by various network requests, including web crawlers, CC attacks, and scanning requests. Each user's website system has a fixed request capacity limit. When the number of requests a website receives reaches this limit, it will be unable to provide normal service, severely impacting the user's business operations. Limited by the cost of website construction, users cannot perform unlimited hardware expansion or architecture upgrades to their website system after it reaches its load limit. Therefore, existing technologies lack effective means to predict and perceive website load risks before network traffic reaches the system and to implement proactive keep-alive measures. This makes websites susceptible to service paralysis due to overload, resulting in damage to user experience and business revenue. Summary of the Invention

[0003] To address the aforementioned technical issues, embodiments of the present invention provide an intelligent website keep-alive method, apparatus, and storage device. This method pre-processes website requests at cloud protection nodes, accurately predicts website load risks through offline modeling and real-time analysis, and then keeps the website alive through tiered intelligent handling. This eliminates the need for unlimited expansion of the website system, effectively reducing costs and ensuring the website continues to provide normal external services.

[0004] In a first aspect, embodiments of the present invention provide a method for keeping a smart website alive, the method comprising:

[0005] Using the website's historical logs as input, offline data processing is performed, following the data processing flow of data selection, data cleaning, data exploration, data parsing, feature standardization, dataset partitioning, data modeling, and model evaluation, to complete the entire data processing process and build a website model;

[0006] Real-time data processing is performed on the website's real-time request data. Combined with the website model built in the offline data processing stage, the maximum number of requests that the website can withstand under different configuration thresholds is deduced in reverse, the current load risk of the website is calculated, and the load risk coefficient is determined.

[0007] Based on the load risk coefficient obtained during the real-time data processing phase, corresponding intelligent keep-alive actions are executed, and intelligent reminders are provided to the cloud protection operation terminal and user terminal.

[0008] Optionally, the website model includes three main models: a 502 risk prediction model, a TCP connection establishment time anomaly attribution model, and a first byte protocol time anomaly prediction model.

[0009] Optionally, the data parsing specifically includes: statistically analyzing the raw log data and constructing data features based on the potential data relationships identified in the data exploration phase; then, statistically analyzing the top 20 URLs without request parameters with the longest first-byte response time on the user's website, and setting these URLs with complex and time-consuming processing logic as the slow URL set M; next, statistically analyzing multi-dimensional feature data; and finally, filtering out minute-by-minute data that did not contain 502 errors or had fewer than 100 502 errors to ensure the correlation between feature data and website load risk.

[0010] Optionally, multi-dimensional feature data can be collected, including: total number of requests per minute, total number of 502 errors per minute, average and deviation of TCP connection establishment time per minute, average and deviation of first byte response time per minute, number of requests for each URL in set M and total number of requests for all URLs, average and deviation of first byte response time for each URL in set M, average and deviation of first byte response time for all URLs, and average and deviation of first byte response time for all API requests.

[0011] Optionally, by combining the website model built in the offline data processing stage, the maximum number of requests that the website can withstand under different configuration thresholds can be deduced in reverse. Specifically, these include: the maximum number of requests that the website can withstand Q1 when the number of 502 errors is between 100 and 1000; the maximum number of requests that the website can withstand Q2 when the TCP connection establishment time is between 1 and 3 seconds; and the maximum number of requests that the website can withstand Q3 when the first byte response time is between 0.5 and 1 second.

[0012] Optionally, the load risk coefficient is determined, specifically including: taking the maximum value among Q1, Q2, and Q3 as the current safe load limit benchmark value of the website, denoted as L, i.e., L=max(Q1,Q2,Q3); then statistically obtaining the actual request volume C within the current time window, calculating the difference between C and L and converting it into a percentage form to obtain the load risk coefficient P, calculated by the formula P=(LC) / L*100%, which intuitively reflects the current load risk level of the website.

[0013] Optionally, intelligent keep-alive actions include: determining the website's load status and performing differentiated actions based on different ranges of load risk coefficients.

[0014] Optionally, differentiated handling actions may be performed, specifically including:

[0015] If the load risk coefficient is 30% or higher, the origin server system is considered to be under normal load and has no service pressure. No special keep-alive actions are required, and the website maintains normal request forwarding and service status.

[0016] If the load risk coefficient is between 10% and 20%, it is determined that the current origin server system is under high load and there is significant pressure on external services. It is necessary to implement rate limiting for real-time requests to the website to reduce the load on the origin server.

[0017] If the load risk coefficient is 10% or below, it is determined that the current website system has too high a load risk and can no longer provide services to the outside world normally. At this time, within the preset threshold time, the cloud protection node will no longer forward subsequent user requests to the origin server, but will call the website mirror traffic that has been crawled in advance, or use CDN static cache resources to provide services to the outside world.

[0018] Secondly, embodiments of the present invention provide an intelligent website keep-alive device for implementing the above-mentioned intelligent website keep-alive method. The intelligent website keep-alive device includes an offline data processing module, a real-time data processing module, and an intelligent handling module. The modules are interconnected and work together to achieve intelligent website keep-alive.

[0019] The offline data processing module is used to perform offline data processing with the website's historical logs as input. It follows the data processing flow of data selection, data cleaning, data exploration, data parsing, feature standardization, dataset partitioning, data modeling, and model evaluation to complete the entire data processing process and build a website model.

[0020] The real-time data processing module communicates with the offline data processing module and is used to perform real-time data processing on the website's real-time request data. Combining the website model built in the offline data processing stage, it reverse-engineers the maximum number of requests the website can withstand under different configuration thresholds, completes the calculation of the website's current load risk, and determines the load risk coefficient.

[0021] The intelligent handling module communicates with the real-time data processing module. Based on the load risk coefficient obtained during the real-time data processing stage, it executes corresponding intelligent keep-alive actions and provides intelligent reminders to the cloud protection operation terminal and user terminal.

[0022] Thirdly, embodiments of the present invention provide a storage device, including a memory and a processor, wherein the processor includes multiple cores, and the memory stores a computer program, which, when executed by the processor, implements the above-mentioned intelligent website keep-alive method.

[0023] This invention provides a method, apparatus, and storage device for keeping a smart website alive. The technical solutions provided by the embodiments of this invention bring at least the following beneficial effects:

[0024] (1) Reduce website construction and maintenance costs. This invention does not require unlimited expansion and upgrade of the website source system. It achieves website survival through cloud protection node pre-limiting, cache scheduling and other means, effectively controlling the hardware and architecture construction costs of the website. At the same time, it provides clear guidance for users' website maintenance by generating business interpretation and maintenance suggestions through large language model, reducing the technical threshold and manpower cost of maintenance.

[0025] (2) Improve the interpretability of the model and the handling: Input the results of offline modeling and the relevant data of real-time handling into the large model to generate natural language interpretation and summary results, so that both technical personnel and non-technical users can clearly understand the load status, risk causes and handling measures of the website, which solves the "black box" problem of traditional algorithm models and improves the practicality and implementation of the technical solution.

[0026] (3) Strong adaptability and easy deployment: The method and device of the present invention can be applied to cloud Waf without modifying the user's website source system. It can quickly provide keep-alive services for user websites connected to cloud protection. It has good adaptability to both existing and newly connected websites, and is easy to deploy and has low modification costs.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating the method for keeping a smart website alive;

[0030] Figure 2 A schematic diagram of the components of an intelligent website keep-alive device. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] Before describing the technical solution of the present invention in detail, the technical background and technical terms involved in the technical solution will be explained first:

[0033] Cloud WAF (Cloud Web Application Firewall) is a cloud-based web application security product, a core protection measure against web application layer attacks within a network security protection system. Unlike traditional hardware / software WAFs, it eliminates the need for local hardware deployment or software installation by enterprises, providing security protection for websites, API interfaces, mini-programs, and other web applications through a cloud service model. The intelligent website keep-alive solution proposed in this invention is directly deployed on the cloud protection node of Cloud WAF. Leveraging Cloud WAF's traffic redirection, log collection, and request forwarding control capabilities, it performs pre-load risk analysis and keep-alive handling of user website requests without requiring modifications to the user's origin server system. This represents a significant upgrade and supplement to Cloud WAF's existing traffic control and security protection capabilities.

[0034] Figure 1 A flowchart illustrating the method for keeping a smart website alive. This method, applied to cloud-based web applications (WAF), comprises three core steps: offline data processing, real-time data processing, and intelligent handling, as detailed below:

[0035] Step S101: Using the website's historical logs as input, perform offline data processing, following the data processing flow of data selection, data cleaning, data exploration, data parsing, feature standardization, dataset partitioning, data modeling, and model evaluation to complete the entire data processing process and build a website model.

[0036] The core of the above steps is to complete the entire data processing process and build a website model through an offline data processing engine. The overall process follows a typical data processing flow: data selection, data cleaning, data exploration, data processing, feature standardization, dataset partitioning, data modeling, and model evaluation. The final output website model includes three main models: a 502 risk prediction model, a TCP connection establishment time anomaly attribution model, and a first-byte protocol time anomaly prediction model. The specific sub-steps are as follows:

[0037] (1) Data selection: For user websites that are already in normal operation, select their business data for the most recent 10 days as the basic data; for user websites that have just been connected to cloud protection, select their business data for all recent working days as the basic data to ensure the validity and representativeness of the data.

[0038] (2) Data cleaning: Filtering out abnormal and useless data in the original data, specifically including removing log data with missing fields, and removing log data that the cloud protection node has not forwarded to the user's website server, to ensure the integrity and authenticity of the data processed later.

[0039] (3) Data exploration: In order to understand the distribution pattern of the data, data during peak business hours on weekdays, such as website data at 10-11 am and 2-3 pm, are selected. Through visual descriptive statistics methods such as histograms, scatter plots, group statistics, and pivot tables, the correlation and internal patterns of user website data are preliminarily analyzed and mined.

[0040] (4) Data Analysis: The original log data is in the form of a single log corresponding to a single request. In this step, the original log data is first statistically analyzed and data features are constructed according to the potential data relationships identified in the data exploration phase. Then, the top 20 URLs with the longest first-byte response time in the user website without request parameters are statistically analyzed. The URLs with complex processing logic and time consumption in this type of origin site are set as the slow URL set M. Next, multi-dimensional feature data are statistically analyzed, including the total number of requests per minute, the total number of 502 errors per minute, the average and deviation of TCP connection establishment time per minute, the average and deviation of first-byte response time per minute, the number of requests for each URL in set M and the total number of requests for all URLs, the average and deviation of first-byte response time for each URL in set M, the average and deviation of first-byte response time for all URLs, and the average and deviation of first-byte response time for all API requests. Finally, minute data that does not have 502 errors or has less than 100 502 errors are filtered out to ensure the correlation between feature data and website load risk.

[0041] (5) Feature standardization: In order to avoid the impact of the difference in the numerical range of data of each feature dimension on the model construction, the normalization formula (x-min) / (max-min) is used to process the feature data of each dimension, and all feature data are uniformly scaled to the interval [0,1] to achieve feature data standardization.

[0042] (6) Dataset partitioning: The processed standardized data is partitioned using K-fold cross-validation. In this invention, K is 5, which means the dataset is divided into 5 equal parts. Four parts are selected in turn as training data and one part is selected as evaluation and validation data. The training and validation process is repeated 5 times. The average of the 5 performance evaluation results is taken as the final model evaluation result to improve the reliability of the model.

[0043] (7) Data modeling and model evaluation: The goal of this step is to build the correlation between feature dimensions and to build a 502 risk prediction model, a TCP connection time anomaly attribution model, and a first byte protocol time anomaly prediction model. The TCP connection time anomaly attribution model is used to analyze the feature dimension most related to TCP connection time, and the first byte protocol time anomaly prediction model is used to predict the first byte response time and as the standard for anomaly judgment.

[0044] During modeling, multiple algorithms, including Ridge, RandomForest, and XGBoost, were used for comparative training. RMSE (Root Mean Square Error) and R² (Coefficient of Determination) were used as model performance evaluation metrics to select the optimal training algorithm for each target model. After constructing the optimal model, the feature importance, prediction results, and related rules output by the model were input into a large-scale language model (LLM) using specially constructed prompt words. The LLM then generated a natural language business interpretation of the feature relationships, as well as preliminary suggestions for website operation and expansion, improving the model's interpretability and action guidance, and providing a reference for subsequent website operation and maintenance for users.

[0045] Step S102: Perform real-time data processing on the website's real-time request data. Combine the website model built in the offline data processing stage to deduce the maximum number of requests the website can withstand under different configuration thresholds, complete the calculation of the website's current load risk, and determine the load risk coefficient.

[0046] The core of the above steps is to perform real-time data processing, which specifically includes:

[0047] The log collection component collects log data generated by external access to the website in real time and stores the collected log data in a unified message storage queue, realizing the temporary storage and unified management of real-time data.

[0048] A real-time big data computing framework is used to consume data from the message storage queue in real time. A statistical time window is set every minute according to the consumption time, and the real-time data within the window is statistically analyzed.

[0049] Based on the feature dimension N determined in the offline data processing stage, the statistical data within the minute-level window is statistically aggregated according to the feature dimension, and the feature data is normalized according to the same normalization method to ensure the adaptability of real-time data and offline model.

[0050] Based on the website model built during the offline data processing phase, the maximum number of requests the website can withstand under different configuration thresholds is deduced in reverse. Specifically, these include: the maximum number of requests the website can withstand Q1 when the number of 502 errors is between 100 and 1000; the maximum number of requests the website can withstand Q2 when the TCP connection establishment time is between 1 and 3 seconds; and the maximum number of requests the website can withstand Q3 when the first byte response time is between 0.5 and 1 second.

[0051] Take the maximum value among Q1, Q2, and Q3 as the baseline value of the current website's safe load limit, denoted as L, i.e., L=max(Q1,Q2,Q3); then obtain the actual request volume C within the current time window, calculate the difference between C and L and convert it into a percentage to obtain the load risk coefficient P, calculated by the formula P=(LC) / L*100%. This coefficient directly reflects the current load risk level of the website.

[0052] Step S103: Based on the load risk coefficient obtained in the real-time data processing stage, execute the corresponding intelligent keep-alive action and complete the intelligent reminder to the cloud protection operation terminal and user terminal.

[0053] The above steps include two parts: intelligent keep-alive processing and intelligent alerts.

[0054] (1) Intelligent keep-alive handling: Based on different value ranges of the load risk coefficient P, determine the load status of the website and execute differentiated handling actions:

[0055] ① If P is 30% or higher, it is determined that the current load of the origin server system is normal and there is no service pressure. No special keep-alive actions need to be performed and the website maintains normal request forwarding and service status.

[0056] ② If P is between 10% and 20%, it is determined that the current origin system load is high and there is significant pressure on external services. It is necessary to limit the real-time requests to the website. Specifically, this is achieved through global origin intelligent control. Based on the safe load limit benchmark value L, a reasonable origin QPS is calculated. The cloud protection node limits the number of requests forwarded to the user's website server to no more than 0.9 times the origin QPS. The origin load is reduced by limiting the rate.

[0057] ③ If P is 10% or below, it is determined that the current website system has too high a load risk and can no longer provide services to the outside world normally. At this time, within the preset threshold time (such as 5 minutes), the cloud protection node will no longer forward subsequent user requests to the origin server, but will call the website mirror traffic that has been crawled in advance, or use CDN static cache resources to provide services to the outside world, so as to ensure the user's access experience and realize the website's survival.

[0058] (2) Intelligent Alerts: For website domains requiring intelligent handling actions such as rate limiting and mirroring / caching services, the system extracts characteristic data at the current time, the calculated baseline value of the security load limit L, the load risk coefficient P, and relevant information such as the actual intelligent handling actions performed. This information is input into the general model, and through built-in prompts and output templates, the general model intelligently summarizes the relevant information, outputting more interpretable handling results and risk analysis. Finally, through a preset alarm channel, the summary results are pushed to the cloud protection operation terminal and the user terminal, allowing operators and users to promptly grasp the website's load status and handling situation.

[0059] Figure 2 This is a schematic diagram of the intelligent website keep-alive device. This device, used in a cloud-based web application (WAF), includes an offline data processing module, a real-time data processing module, and an intelligent handling module. These modules communicate and work together to achieve intelligent website keep-alive.

[0060] The offline data processing module is used to perform offline data processing with the website's historical logs as input. It follows the data processing flow of data selection, data cleaning, data exploration, data parsing, feature standardization, dataset partitioning, data modeling, and model evaluation to complete the entire data processing process and build a website model.

[0061] The real-time data processing module communicates with the offline data processing module and is used to perform real-time data processing on the website's real-time request data. Combining the website model built in the offline data processing stage, it reverse-engineers the maximum number of requests the website can withstand under different configuration thresholds, completes the calculation of the website's current load risk, and determines the load risk coefficient.

[0062] The intelligent handling module communicates with the real-time data processing module. Based on the load risk coefficient obtained during the real-time data processing stage, it executes corresponding intelligent keep-alive actions and provides intelligent reminders to the cloud protection operation terminal and user terminal.

[0063] The offline data processing module includes a data selection unit, a data cleaning unit, a data exploration unit, a data parsing unit, a feature standardization unit, a dataset partitioning unit, and a modeling and evaluation unit.

[0064] The data selection unit is used to perform data selection operations, filtering out website business data that meets the requirements, and providing a basis for subsequent processing.

[0065] The data cleaning unit, connected to the data selection unit, is used to receive the filtered basic data, perform data cleaning operations, and filter out abnormal and useless data.

[0066] The data exploration unit, connected to the data cleaning unit, is used to receive cleaned data and analyze its distribution, relationships, and patterns through visualization.

[0067] The data parsing unit, connected to the data exploration unit, is used to receive the results after data exploration, perform feature construction on the original logs, statistical analysis of slow URL sets, statistical analysis of multi-dimensional feature data, and filter valid minute data points.

[0068] The feature standardization unit, connected to the data processing unit, is used to receive the processed feature data and perform normalization processing to achieve feature standardization.

[0069] The dataset partitioning unit, connected to the feature standardization unit, is used to receive standardized feature data and complete the dataset partitioning using cross-validation.

[0070] The modeling and evaluation unit, connected to the dataset partitioning unit, receives the partitioned training and validation data and constructs a 502 risk prediction model, a TCP connection time anomaly attribution model, and a first-byte protocol time anomaly prediction model, respectively. The optimal algorithm is selected through multi-algorithm comparison training, and the model-related information is input into a large language model to generate business interpretation and operation and maintenance suggestions. At the same time, the final website model is sent to the real-time data processing module.

[0071] The real-time data processing module collects real-time request log data from the website, performs minute-level statistical analysis and feature processing, and, in conjunction with the website model sent by the offline data processing module, calculates the safety load limit benchmark value L and the load risk coefficient P. It then sends P and related statistical data to the intelligent handling module. This module includes a log collection unit, a message queue unit, a real-time calculation unit, a feature processing unit, and a load calculation unit.

[0072] The log collection unit collects log data generated by external access to the website in real time.

[0073] The message queue unit is electrically connected to the log collection unit, receives and temporarily stores the real-time log data collected by the log collection unit, and realizes unified data management.

[0074] The real-time computing unit is electrically connected to the message queue unit. It uses a real-time big data computing framework to consume data from the message queue in real time and completes data statistics in a time window of one minute.

[0075] The feature processing unit is electrically connected to the modeling and evaluation unit of the real-time computing unit and the offline data processing module. It receives real-time statistical data and feature dimension information of the offline model, and completes data aggregation and normalization processing according to preset feature dimensions.

[0076] The load calculation unit is electrically connected to the feature processing unit. It receives the processed real-time feature data, combines it with the website model to deduce the maximum request volume under multiple thresholds, calculates the safe load upper limit benchmark value L and the load risk coefficient P, and sends the relevant data to the intelligent handling module.

[0077] The intelligent handling module executes corresponding tiered keep-alive actions based on the load risk coefficient P sent by the real-time data processing module, and intelligently summarizes the relevant data through a general large model. Finally, it sends intelligent alerts to the cloud protection operation terminal and the user terminal via an alarm channel. This module includes a tiered handling unit, a data aggregation unit, a large model processing unit, and an alarm push unit.

[0078] The tiered handling unit is connected to the load calculation unit of the real-time data processing module. It receives the load risk coefficient P, determines the website load status based on the value range of P, and performs tiered handling actions such as normal monitoring, request rate limiting, and mirror / caching services.

[0079] The data aggregation unit is connected to the hierarchical processing unit and the load calculation unit. It extracts and aggregates the characteristic data, the safe load upper limit benchmark value L, the load risk coefficient P, and the actual processing actions performed at the current time point.

[0080] The large model processing unit connects to the data aggregation unit, receives the aggregated data, and combines the built-in prompts and output templates to intelligently summarize the data and generate interpretable results.

[0081] The alarm push unit connects to the large model processing unit, receives the summary results generated by the large model, and pushes the results to the cloud protection operation terminal and user terminal through the preset alarm channel to realize intelligent reminders.

[0082] The aforementioned device is used to implement the aforementioned intelligent website keep-alive method. Each module included in the device is used to implement the corresponding steps in the aforementioned intelligent website keep-alive method. For details, please refer to the corresponding parts of the aforementioned method, which will not be repeated here.

[0083] In summary, this invention preprocesses website requests at cloud protection nodes. Through offline data modeling and real-time data analysis, it accurately predicts load risks before network traffic reaches the website's origin system, solving the problem of existing technologies' inability to detect website load risks in advance and enabling early detection and handling of risks. Based on different ranges of load risk coefficients, it executes differentiated actions such as normal monitoring, request rate limiting, and mirroring / caching services. Adaptive keep-alive strategies are adopted for different load states, avoiding resource waste caused by over-processing while quickly ensuring website service continuity under high-risk conditions, achieving intelligent website keep-alive.

[0084] The above solution will be illustrated below through a specific embodiment and in conjunction with a specific application scenario.

[0085] An e-commerce company built an online retail website and integrated a cloud service provider's cloud WAF for basic security protection. The website experiences approximately 30,000 requests per second (QPS) daily. During peak sales periods, it is vulnerable to web crawlers scraping product information in bulk and malicious DDoS attacks combined with normal user access, leading to overload of the origin server, resulting in 502 errors, slow page response, and even service paralysis. Limited by server expansion costs, the company cannot indefinitely increase the origin server's capacity. Therefore, it deployed the intelligent website keep-alive solution of this invention in the cloud WAF to achieve proactive load risk detection and intelligent keep-alive, ensuring normal service of the website, especially during peak sales periods. The specific implementation process is as follows:

[0086] Step 1: Offline data processing to build a website-specific model

[0087] The e-commerce website is already in operation. The solution first activates the offline data processing module, selecting the website's full access history logs for the most recent 10 days (including approximately 8 million request logs) as the basic data. The system then completes full-dimensional processing and builds the website model according to the established workflow.

[0088] Data cleaning: 32,000 log entries with missing fields were removed, along with 156,000 maliciously intercepted log entries that were not forwarded to the origin server by Cloud WAF, ensuring data validity.

[0089] Data exploration: By selecting peak business data from 10-11 am, 2-3 pm and 8-9 pm on weekdays, and through visualization analysis, it was found that when the number of 502 errors per minute exceeds 100 and the average TCP connection establishment time exceeds 1.2 seconds, the load on the origin server is approaching the critical value.

[0090] Data processing: The 20 slow URLs with the longest first byte response were identified (mostly product detail pages and order query interfaces, forming a slow URL set M). The characteristic data statistics of 12 dimensions, such as total requests per minute, total number of 502 errors, and average / deviation of TCP connection establishment time, were completed, and minute-by-minute data with fewer than 100 502 errors were filtered out.

[0091] Feature standardization and dataset partitioning: All feature data are scaled to the [0,1] interval by (x-min) / (max-min), and the data is divided into training and validation sets by 5-fold cross-validation.

[0092] Modeling and Evaluation: Three models were constructed for 502 risk prediction, TCP connection time anomaly attribution, and first byte protocol time anomaly prediction. After comparative training with Ridge, RandomForest, and XGBoost algorithms, XGBoost was selected as the optimal algorithm (lowest RMSE value and R² closest to 1). The importance of model features and prediction results were input into the large model to obtain the business interpretation that "when the proportion of URL requests in set M exceeds 30%, the average first byte response time will exceed 0.8s", and the operation and maintenance suggestions that "optimize the caching of the order query interface and limit the frequency of crawler access to the product details page".

[0093] Step 2: Real-time data processing and calculation of load risk coefficient.

[0094] After the promotional event begins, the solution activates the real-time data processing module to perform minute-level analysis of real-time access requests to the official website. This analysis, combined with the offline-built website model, calculates the load risk coefficient. Specific operations include:

[0095] Real-time data collection and statistics: The log collection component collects access logs from the official website in real time and stores them in a Kafka message queue. The Flink real-time big data computing framework consumes the data from the queue and performs data statistics in a time window of one minute.

[0096] Feature processing: Based on the 12 feature dimensions determined in the offline phase, minute-level data is statistically aggregated and normalized to ensure compatibility with the offline model.

[0097] Load threshold and risk factor calculation: Combining the website model, we reverse-engineer the maximum number of requests that the origin server can withstand under multiple thresholds: Q1 = 8000 qps when the number of 502 errors exceeds 500, Q2 = 7500 qps when the TCP connection establishment time exceeds 2 seconds, and Q3 = 7800 qps when the first byte response time exceeds 1 second. We take the maximum value L = 8000 qps as the baseline value for the safe load limit.

[0098] If the actual real-time request volume of the official website is C=6800 qps within a certain minute window, the load risk coefficient is calculated as P=(8000-6800) / 8000×100%=15%.

[0099] During peak promotional periods, the actual request volume within a certain minute window is C=7440 qps, and the calculated value is P=(8000-7440) / 8000×100%=7%.

[0100] Step 3: Intelligent handling, implementing tiered keep-alive measures + intelligent alerts

[0101] The intelligent handling module executes differentiated keep-alive actions based on the real-time calculated load risk coefficient P, and simultaneously provides intelligent alerts. The handling for the two peak scenarios mentioned above is as follows:

[0102] Scenario 1: P = 15% (10% < P < 30%), the source station has a high load

[0103] Hierarchical handling: It is determined that there is pressure on the external service of the source station, and flow limiting handling is initiated. Based on the safety load upper limit reference value L = 8000 qps, the return-to-source qps = 8000 is calculated. The cloud Waf node limits the request volume forwarded to the source station within 8000 × 0.9 = 7200 qps. The excess legitimate requests are queued for forwarding, and malicious crawler / attack requests are directly intercepted in the cloud to reduce the load of the source station.

[0104] Intelligent reminder: The data summary unit extracts the characteristic data of this time window (the average TCP connection establishment time is 1.1 s, the number of 502 errors is 180, the proportion of URL requests in set M is 28%), L = 8000 qps, P = 15%, and the handling action of "flow limiting to 7200 qps". After inputting into the general large model, the summary result "During the current major promotion of the e-commerce official website, the load is high, the number of 502 errors reaches 180, flow limiting handling has been executed, the source station forwarding requests are limited to 7200 qps, and it is recommended to pay attention to the response speed of the product details page interface" is generated, and it is pushed to the e-commerce enterprise operation and maintenance team and the cloud protection operation side through the enterprise WeChat and SMS alarm channels.

[0105] Scenario 2: P = 7% (P ≤ 10%), the source station load risk is too high

[0106] Hierarchical handling: It is determined that the source station is approaching the bearing limit and cannot provide external services normally, and the fallback keep-alive handling is initiated. Within the preset 5-minute threshold time, the cloud Waf node stops forwarding subsequent requests to the source station server, and calls the static page mirror traffic of the official website products crawled in advance + CDN static cache (product pictures, home page, category pages, etc.) to provide access services for users, ensuring that users can normally browse product information. Only a small number of core order submission requests are queued for forwarding to the source station to minimize the pressure on the source station.

[0107] Intelligent reminder: The large model generates the summary result "During the peak load of the current e-commerce official website major promotion, the risk is extremely high. The actual request volume has approached the bearing limit of the source station. The fallback service of CDN cache + mirror traffic has been started. The source station forwarding of non-core requests will be suspended within 5 minutes. It is recommended that the operation and maintenance personnel urgently check the status of the source station server" for this scenario, and it is pushed through the alarm channel. At the same time, the cloud protection operation side synchronously intervenes to assist the enterprise in monitoring the load change of the source station.

[0108] Fourth step: Recovery process after risk解除

[0109] After the 5-minute buffer period ended, the real-time data processing module detected that the real-time request volume of the official website dropped back to C=6000 qps, and calculated P=25%. The intelligent handling module immediately stopped the buffer service, restored the normal request forwarding from the cloud Waf node to the origin server, and gradually lifted the rate limit. At the same time, it pushed a reminder to the operation and maintenance team that "the origin server load risk has decreased and normal request forwarding has been restored", ensuring a smooth transition of the official website service.

[0110] During this major promotional period, by deploying this intelligent website keep-alive solution in the cloud WAF, the e-commerce website achieved proactive risk detection and tiered handling of load risks, and no service outages occurred throughout the entire process.

[0111] When the load is high, precise rate limiting keeps the origin server load within a safe range, reducing the number of 502 errors by 78% and the average page response time from 1.5s to 0.8s.

[0112] When the load risk is extremely high, CDN caching and mirror traffic are used as a backup to maintain the official website access success rate of over 99%, avoiding order loss due to service failure.

[0113] Without the need for unlimited expansion of the origin server, service continuity during major promotional events is ensured solely through traffic control and intelligent keep-alive mechanisms on the cloud WAF platform. This directly reduces server expansion costs by approximately 60%. At the same time, the business interpretation and operation and maintenance suggestions generated by the large model provide clear direction for enterprises' subsequent website interface optimization and traffic control.

[0114] The solution of this invention is directly deployed on a cloud-based web application (WAF). Relying on the core capabilities of cloud WAF in traffic redirection, log collection, and request forwarding control, it achieves the goal of "risks being detected before traffic reaches the origin server, and handling being implemented first." No modifications are required to the origin server system of e-commerce enterprises; it can be quickly integrated simply through cloud WAF configuration. This not only supplements the original security protection capabilities of cloud WAF but also upgrades the traffic management capabilities of cloud WAF to an integrated capability of "intelligent load control + keep-alive," perfectly adapting to the traffic protection needs of various web applications such as e-commerce, government affairs, and corporate websites.

[0115] According to embodiments of the present invention, a storage device is also provided. The storage device may include a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, it can implement the methods provided in any of the above embodiments, and its execution method and beneficial effects are similar, and will not be described again here. In addition, depending on the specific application, the storage device may also include any other suitable components.

[0116] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. The same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on describing the differences from other embodiments, and relevant parts can be referred to the descriptions of other method embodiments.

[0117] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0118] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An intelligent website keep-alive method, characterized in that, The method comprises: Performing offline data processing with historical logs of the website as input, following the data processing procedures of data selection, data cleaning, data exploration, data analysis, feature standardization, data set division, data modeling and model evaluation, completing the whole-process processing of data and building a website model; Performing real-time data processing on real-time request data of the website, combining the website model built in the offline data processing stage, reversely deducing the maximum request amount that the website can bear under different configuration thresholds, completing the calculation of the current load risk of the website, and determining the load risk coefficient; Based on the load risk coefficient obtained in the real-time data processing stage, performing corresponding intelligent keep-alive disposal actions, and completing intelligent reminders to the cloud protection operation end and the user end.

2. The intelligent website keep-alive method of claim 1, wherein, The website model includes three models: a 502 risk prediction model, a TCP connection time anomaly attribution model, and a first byte protocol time anomaly prediction model.

3. The intelligent website keep-alive method of claim 1, wherein, The data analysis specifically includes: according to the potential data relationships identified in the data exploration stage, performing statistics on the original log data and constructing data features; then, statistics are performed on the top 20 URLs without request parameters in the user website with the longest first byte response time, and the URLs with complex and time-consuming source station processing logic are set as a slow URL set M; then, multi-dimensional feature data is counted; finally, the data points with no 502 error or less than 100 502 errors are filtered out to ensure the relevance of the feature data and the website load risk.

4. The intelligent website keep-alive method of claim 4, wherein, The multi-dimensional feature data specifically includes: total number of requests per minute, total number of 502 errors per minute, mean and deviation of TCP connection time per minute, mean and deviation of first byte response time per minute, number of requests for each URL in set M and total number of requests for all URLs, mean and deviation of first byte response time for each URL in set M, mean and deviation of first byte response time for all URLs, and mean and deviation of first byte response time for all API requests.

5. The intelligent website keep-alive method of claim 1, wherein, In combination with the website model built in the offline data processing stage, the maximum request amount that the website can bear under different configuration thresholds is reversely deduced, specifically including: the maximum request amount Q1 that the website can bear under a specified number of 502 errors between 100 and 1000; the maximum request amount Q2 that the website can bear under a specified threshold of TCP connection time between 1 and 3 seconds; and the maximum request amount Q3 that the website can bear under a specified threshold of first byte response time between 0.5 and 1 second.

6. The intelligent website keep-alive method of claim 5, wherein, Determining the load risk coefficient specifically includes: taking the maximum value of Q1, Q2 and Q3 as the upper limit reference value of the safe load of the current website, denoted as L, i.e. L = max(Q1, Q2, Q3); then, the actual request amount C in the current time window is obtained, the difference between C and L is calculated and converted into a percentage form to obtain the load risk coefficient P, and the calculation formula is P = (L-C) / L*100%, which directly reflects the current load risk degree of the website.

7. The intelligent website keep-alive method of claim 1, wherein, Intelligent keep-alive disposal actions specifically include: determining the load state of the website according to different numerical intervals of the load risk coefficient and performing differentiated disposal actions.

8. The intelligent website keep-alive method of claim 1, wherein, Performing differentiated disposal actions specifically includes: If the load risk coefficient is 30% or higher, the origin server system is considered to be under normal load and has no service pressure. No special keep-alive actions are required, and the website maintains normal request forwarding and service status. If the load risk coefficient is between 10% and 20%, it is determined that the current origin system is under high load and there is significant pressure on external services. It is necessary to implement rate limiting for real-time requests to the website to reduce the load on the origin. If the load risk coefficient is 10% or below, it is determined that the current website system has too high a load risk and can no longer provide services to the outside world normally. At this time, within the preset threshold time, the cloud protection node will no longer forward subsequent user requests to the origin server, but will call the website mirror traffic that has been crawled in advance, or use CDN static cache resources to provide services to the outside world.

9. An intelligent website keep-alive device, which is used to implement the intelligent website keep-alive method in any one of claims 1 to 8, characterized in that, The intelligent website survival device includes an offline data processing module, a real-time data processing module, and an intelligent handling module. The modules are interconnected and work together to achieve intelligent website survival. The offline data processing module is used to perform offline data processing with the website's historical logs as input. It follows the data processing flow of data selection, data cleaning, data exploration, data parsing, feature standardization, dataset partitioning, data modeling, and model evaluation to complete the entire data processing process and build a website model. The real-time data processing module communicates with the offline data processing module and is used to perform real-time data processing on the website's real-time request data. Combining the website model built in the offline data processing stage, it reverse-engineers the maximum number of requests the website can withstand under different configuration thresholds, completes the calculation of the website's current load risk, and determines the load risk coefficient. The intelligent handling module communicates with the real-time data processing module. Based on the load risk coefficient obtained during the real-time data processing stage, it executes corresponding intelligent keep-alive actions and provides intelligent reminders to the cloud protection operation terminal and user terminal.

10. A storage device, comprising: include: A memory and a processor, the processor including multiple cores, wherein the memory stores a computer program that, when executed by the processor, implements the intelligent website keep-alive method as described in any one of claims 1 to 8.

Images