Intelligent cache preheating method and system based on historical access data

Abstract

The invention discloses an intelligent cache preheating method and system based on historical access data, and aims to solve the problems of insufficient prediction capability, low resource utilization rate and the like of a traditional cache mechanism. The method comprises the following steps: firstly, carrying out multi-dimensional data acquisition and distributed storage; extracting time, user, resource and system multi-dimensional features through data preprocessing and feature engineering; deeply analyzing and mining a time sequence rule, a user behavior cluster and a resource association rule through an access mode; then, a mixed prediction model fusing LSTM-ARIMA is constructed, and prediction precision is improved in combination with transfer learning and an online updating mechanism; a dynamic preheating strategy is generated based on a prediction result, and efficient preheating is achieved through high-probability resource screening, multi-objective optimization and batch scheduling; and finally, the effect is continuously improved through real-time monitoring, a fusing mechanism and closed-loop optimization. The method can improve the cache hit rate and reduce the system response time, and is suitable for distributed systems such as e-commerce and social networks.

Description

Technical Field

[0001] This invention relates to the field of data access performance optimization technology, specifically to an intelligent cache preheating method and system based on historical access data. Background Technology

[0002] In the field of modern internet technology, caching, as a core means to improve system performance and reduce the pressure on backend database access, is widely used in various distributed systems. Especially in scenarios such as e-commerce platforms, social networks, and online services, caching mechanisms directly affect the system's response speed and stability. By temporarily storing frequently accessed data in a cache layer, such as Redis or Memcached, direct queries to backend storage can be significantly reduced, thereby shortening the response time for user requests and improving service capacity.

[0003] However, in scenarios with high concurrency and drastic traffic fluctuations, such as e-commerce platform promotions, holiday flash sales, or sudden surges in access caused by trending events, traditional caching mechanisms gradually reveal significant limitations, specifically:

[0004] (1) Passive loading leads to cold start problems. The traditional "on-demand caching" strategy only loads data into the cache when a user requests it for the first time. This causes a large number of requests to directly penetrate the database due to cache misses in the early stages of a traffic surge, triggering "cache avalanche" or "cache penetration", resulting in a sharp increase in database pressure, response delays, or even service crashes. For example, within 10 minutes of the start of a promotional activity on an e-commerce platform, the first access requests to the details page of a hot-selling product accounted for more than 60%, directly causing a 300% surge in database queries and increasing the system response time from the normal 50ms to more than 500ms.

[0005] (2) Lack of predictive ability for access patterns. User access behavior shows significant regularity in dimensions such as time, group, and resource association, but traditional caching mechanisms cannot mine these patterns based on historical data, making it difficult to predict high-frequency access resources in the future, resulting in a disconnect between cache resource allocation and actual access needs.

[0006] (3) Low resource utilization and uneven load. During off-peak hours, cache space is often idle due to lack of effective preheating, while during peak hours, cache misses lead to resource waste and insufficient resources. At the same time, manually performing cache preheating during peak hours will exacerbate system load competition and further reduce service stability.

[0007] Therefore, existing technologies lack a method and system that can integrate multi-dimensional access data, automatically identify access patterns, and dynamically generate preheating strategies. Summary of the Invention

[0008] To address the aforementioned shortcomings of existing technologies, this invention predicts future access trends based on historical access data, loads high-frequency resources into the cache in advance using intelligent methods, and dynamically adapts to system load and cache capacity.

[0009] The first objective of this invention is to propose an intelligent cache preheating method, comprising the following steps:

[0010] S1: Uses a Kafka streaming cluster to access user access requests from all channels, and uses a two-tier storage architecture to store user access requests;

[0011] S2: Standardize the format of access requests, handle outliers, and isolate traffic to construct a multi-dimensional feature matrix of "time-user-resource-system";

[0012] S3: Construct an LSTM-ARIMA model for temporal pattern recognition, cluster user behavior, and generate high-value association rules;

[0013] S4: Construct an attention-weighted Stacking ensemble model, fuse multi-dimensional feature matrices through a hierarchical training mechanism, and optimize the model using online learning;

[0014] S5: Generate a preheated resource list based on access probability ranking and association rules, construct a dynamic batching strategy using a multi-objective optimization algorithm, and adjust the batch interval through exponential backoff;

[0015] S6: Implement a tiered loading mode and a monitoring circuit breaker mechanism, and combine optimization algorithms to fine-tune the parameters of resources for prediction errors.

[0016] Preferably, the dual-layer storage architecture includes HBase columnar storage for storing frequently accessed data and HDFS distributed storage for storing all data.

[0017] Preferably, in the LSTM-ARIMA model, the LSTM layer is set with 128 neurons to capture long-term trends, the ARIMA parameters are determined through correlation analysis, and the output weights of LSTM and ARIMA are adjusted using a dynamic weight fusion strategy.

[0018] Preferably, the clustering of user behavior includes using the K-means++ algorithm to group user feature vectors, prioritizing regions with higher data density when initializing centroids, and evaluating and determining the optimal number of clusters using silhouette coefficients.

[0019] Preferably, the generation of high-value association rules includes using the Apriori algorithm, introducing a time decay factor, combining resource similarity calculated by the Jacobi index to screen effective rules, and establishing a multi-dimensional screening mechanism of support, confidence, and lift to generate high-value association rules.

[0020] Preferably, the Stacking ensemble model includes a base model composed of LSTM, XGBoost, and LightGBM, and a meta-model composed of CatBoost.

[0021] Preferably, the dynamic batching strategy includes, based on the XGBoost load prediction model, taking CPU utilization and remaining memory characteristics as inputs, outputting future load prediction values, and loading or reducing resources according to the load prediction values.

[0022] Preferably, step S6 includes:

[0023] S61: The hierarchical loading mode adopts a differentiated loading strategy based on the different access probabilities of resources;

[0024] S62: Deploy a real-time monitoring system that triggers a circuit breaker mechanism when key indicators reach thresholds. These key indicators include cache hit rate, latency, and database query volume.

[0025] A second objective of this invention is to propose an intelligent cache preheating system, comprising the following modules:

[0026] (1) Data acquisition module: responsible for accessing user access requests from all channels through the Kafka streaming processing cluster, using a two-layer storage architecture;

[0027] (2) Preprocessing module: Performs cleaning operations such as format standardization, outlier handling, and traffic isolation on access requests, and extracts multi-dimensional features of time, user, resource, and system to construct a feature matrix;

[0028] (3) Pattern analysis module: The LSTM-ARIMA hybrid model is used to mine time series patterns, the K-means++ algorithm is used to cluster user behavior, and the Apriori algorithm is used to generate association rules for high-value resources;

[0029] (4) Prediction model module: Construct an attention-weighted Stacking ensemble model, combined with transfer learning and online update mechanism, to improve the prediction accuracy of high-frequency access resources in the future;

[0030] (5) Strategy generation module: Based on access probability ranking and association rules, preheating resources are selected, and dynamic batching strategies are generated through multi-objective optimization algorithms. Combined with load prediction model and exponential backoff strategy, batch size and interval are adjusted to achieve efficient preheating scheduling.

[0031] (6) Execution adjustment module: Implement a graded loading mode and ensure the stability and controllability of the preheating process through a monitoring system and circuit breaker mechanism;

[0032] (7) Evaluation and optimization module: The preheating effect is evaluated through control experiments and multi-dimensional indicators, and the parameters are optimized and adjusted for resources with excessive prediction errors.

[0033] The beneficial effects of this invention are as follows:

[0034] (1) Significantly improve system performance indicators and greatly increase cache hit rate. Through multi-dimensional data prediction and dynamic preheating strategy, the cache hit rate is higher than the passive loading efficiency of traditional methods, effectively reducing the risk of cache penetration and cache avalanche.

[0035] (2) The system response time is significantly reduced. High-frequency resources are preloaded into the cache layer, which reduces the system response time. For example, in e-commerce promotion scenarios, the response time of the product details page can be greatly compressed, improving the user experience.

[0036] (3) The pressure on the database is significantly reduced, a large number of invalid database queries are reduced, the database overload problem during peak traffic periods is avoided, and the stable operation of core services is guaranteed.

[0037] (4) The system load is more balanced. The scheduling mechanism based on load prediction can complete the warm-up operation during low load periods, avoid resource competition during peak periods, and will not impact normal services.

[0038] In summary, this invention achieves intelligent and automated cache preheating, which is of great value in improving system stability, optimizing user experience, and reducing operation and maintenance costs. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the steps of the intelligent cache preheating method according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] This invention provides an intelligent cache preheating method based on historical access data. By constructing a complete closed-loop mechanism encompassing collection, analysis, prediction, execution, evaluation, and optimization, it offers a systematic solution to the shortcomings of traditional cache preheating methods. Figure 1 As shown, the specific steps include the following:

[0043] S1: Multi-dimensional Data Acquisition and Distributed Storage. To achieve efficient capture and reliable storage of data accessed from all channels, this step constructs a full-link architecture covering data access, processing, and storage, implemented as follows:

[0044] S11: Omnichannel Data Access. Deploy a Kafka streaming cluster, an open-source streaming platform with high-throughput log processing capabilities. It can be compatible with multiple transport protocols such as HTTP and TCP through a multi-protocol adapter, thereby handling user access requests from omnichannel sources including apps, mini-programs, and web pages.

[0045] To balance data integrity and processing efficiency, a dynamic sampling mechanism is introduced. When traffic is at its peak, critical requests are sampled using a predetermined retention rate, while duplicate logs are filtered out. This reduces the burden of processing redundant data while ensuring the integrity of core access information.

[0046] To address data quality issues, multi-source data verification rules are configured. When abnormal timestamp formats or missing resource identifiers are detected, the system automatically triggers a repair process. If the repair fails, an alarm is issued immediately or the data is discarded to ensure that data entering subsequent processing stages meets quality standards.

[0047] S12: Distributed storage architecture. It adopts a layered design. High-frequency access data is stored in HBase columnar format, leveraging its columnar storage characteristics to support millisecond-level random read / write operations, meeting the need for fast access to recently hot data. Historical full data is archived to HDFS (Hadoop Distributed File System), and data consistency between distributed nodes is achieved through the RAFT (Replicated and Fault-Tolerant, a consistency protocol in distributed systems), ensuring the reliability of long-term data storage.

[0048] To optimize the load balancing of storage nodes, a hash-based sharding storage strategy is adopted for resource identifiers. Combined with a consistent hashing algorithm, this minimizes the overhead of data migration when storage nodes are added or removed, ensuring that the storage system maintains high efficiency during dynamic changes.

[0049] S2: Data Preprocessing and Feature Engineering. Data preprocessing and feature engineering are fundamental to mining access patterns. Through intelligent cleaning and multi-dimensional feature extraction, raw data is transformed into structured features that can be directly used for model training. The specific process is as follows:

[0050] S21: Intelligent Data Cleaning. A cleansing pipeline based on a rule engine is built. Data format standardization is achieved through regular expression matching, outliers are identified and processed using the 3σ principle, and missing data is filled in using KNN interpolation to ensure data integrity and consistency. For abnormal traffic such as that from web crawlers, a feature fingerprint database is constructed. Abnormal behavior is accurately identified by analyzing indicators such as request frequency and UserAgent similarity, and isolated and stored to avoid interfering with the analysis of normal data.

[0051] S22: Multimodal feature extraction. Construct a feature matrix containing four dimensions: time, user, resource, and system.

[0052] (1) The time characteristics cover hourly segments (0-23), weekday types (weekdays or weekends), and holiday markers (such as Spring Festival, National Day, etc.) to capture the time regularity of access behavior.

[0053] (2) User characteristics include historical purchase frequency, number of active days in the last 30 days, and device type distribution, reflecting the access habits of different user groups.

[0054] (3) Resource characteristics include category popularity ranking, price range, and access volatility in the past 7 days, which depict the popularity and trend of the resource itself.

[0055] (4) System characteristics involve the average CPU utilization and cache hit rate baseline values ​​during the time period, reflecting the impact of system operation status on access.

[0056] For unstructured product categories, the Word2Vec algorithm is used for vectorization, and then the t-SNE algorithm is used to reduce the dimensionality to 16-dimensional feature vectors, which facilitates subsequent model processing.

[0057] S3: In-depth access pattern analysis, through temporal pattern mining, user behavior clustering, and resource association rule extraction, accurately identifies potential access characteristics and patterns, providing a core basis for subsequent prediction models. The specific implementation is as follows:

[0058] S31: Temporal Pattern Recognition. A hybrid LSTM-ARIMA model is constructed, where the LSTM (Long Short-Term Memory) layer has 128 neurons to capture long-term access trends, and the parameters (p, d, q) of the ARIMA (Autoregressive Integrated Moving Average) model are automatically determined through autocorrelation (ACF) and partial autocorrelation (PACF) to adapt to short-term fluctuations. To improve prediction accuracy, a dynamic weight fusion strategy is introduced, adjusting the output weights of LSTM and ARIMA in real time based on the model's prediction error (MAE) over the past 7 days, so that the hybrid model can reflect both long-term trends and adapt to short-term abrupt changes.

[0059] S32: User Behavior Clustering. The K-means++ algorithm is used to group user feature vectors. When initializing centroids, regions with higher data density are prioritized. The optimal number of clusters is evaluated and determined using the silhouette coefficient to ensure the reasonableness of the clustering results. Simultaneously, the system monitors user behavior drift in real time. When the Euclidean distance of user feature vectors exceeds a preset threshold, cluster center updates are automatically triggered to ensure that the clustering results dynamically adapt to changes in user behavior.

[0060] S33: Association Rule Mining. Using the Apriori algorithm, a time decay factor (value 0.8) is introduced to mitigate the influence of historical data. This is combined with resource similarity calculations using the Jacobian index to screen effective rules, improving the timeliness and reliability of the rules. Based on this, a three-dimensional screening mechanism of support, confidence, and lift is established to ultimately output high-value association rules, providing a basis for resource association preheating. For example, "buy a mobile phone → visit a mobile phone case" has a support of 0.15, a confidence of 0.72, and a lift of 1.8.

[0061] S4: Predictive model construction and optimization. Through multi-dimensional feature fusion, ensemble learning strategies, and dynamic update mechanisms, the accuracy of predicting high-frequency access resources in the future is improved, providing a reliable basis for the warm-up strategy. The specific implementation is as follows:

[0062] S41: Feature Fusion and Model Integration. An attention-based weighted fusion layer is designed to dynamically weight time features, user features, resource features, and system features. This dynamic weighting highlights the impact of key features on the prediction results. For example, the weight of time features is set to 0.3, user features to 0.25, resource features to 0.3, and system features to 0.15.

[0063] This embodiment constructs a Stacking ensemble model, which integrates multi-dimensional prediction capabilities through a hierarchical training mechanism. The base model focuses on capturing access patterns across different dimensions, while the meta-model achieves deep integration of multi-source information. The Stacking ensemble model adopts a two-layer training architecture:

[0064] (1) First layer (base model): It consists of three models: LSTM, XGBoost and LightGBM, which learn independently from three dimensions: time series pattern, user behavior clustering and resource association rules and output prediction results respectively.

[0065] (a) LSTM model: to explore the temporal evolution of user access behavior and predict the access frequency and time series distribution of resources in a specific future period.

[0066] Input features: mainly time features weighted by attention mechanism, such as hourly segments, weekday types, holiday identifiers, historical access frequency sequences, etc., while also incorporating some system features, such as average CPU utilization during time periods and baseline cache hit rate.

[0067] Model Design: A 128-neuron LSTM layer is used to capture long-term dependencies through a gating mechanism, such as daily peak access times and traffic fluctuations during periodic promotional activities. The output is a time series sequence of resource access probabilities for each future time period, which is finally aggregated into "access probability for the next 24 hours" as the base model output.

[0068] (b) XGBoost model: Based on user behavior clustering results, predict the probability that a specific resource will be accessed by a certain type of user group.

[0069] Input features: The core features are user characteristics, such as historical purchase frequency, number of active days in the last 30 days, device type, and user cluster identifier, supplemented by resource features, such as category popularity ranking and price range.

[0070] Model design: A tree-structured ensemble is adopted, and regularization is used to control complexity and avoid overfitting. The training objective is "the conditional probability of a resource being accessed by a certain type of user", and the output is the access probability value based on user clustering. For example, the probability of resource A being accessed by a high-frequency active user cluster is 0.75.

[0071] (c) LightGBM model: Based on the association rules between resources, predicts "the probability that when a certain resource is accessed, the target resource will be accessed as well".

[0072] Input features: mainly resource association features, such as support, confidence, lift between resources, resource similarity calculated by Jacobi index, etc., combined with resource characteristics, such as access volatility in the past 7 days.

[0073] Model Design: A decision tree is used to efficiently handle high-dimensional sparse features, focusing on the dynamic updating of association rules. A time decay factor is introduced to weaken the influence of old rules. The output is the probability of associated access based on association rules; for example, when resource B is accessed, the probability of associated access to resource C is 0.68.

[0074] (2) Second layer (meta-model): The CatBoost model is the core, and the output of the base model is used as the input feature. The weights and collaborative relationships of each base model are learned through secondary training, and finally a unified prediction result is output, namely the future access probability of the resource, such as "the comprehensive access probability of the resource in the next 24 hours".

[0075] Input features: The outputs of the base model, namely the temporal access probability of LSTM, the user clustering access probability of XGBoost, and the association rule confidence of LightGBM, form a multi-dimensional probability feature matrix.

[0076] Training objective: To learn the mapping relationship between the outputs of the three base models and the actual access results, and to integrate the advantages by adjusting the weights, such as strengthening the LSTM weights when the temporal regularity is reliable, and strengthening the XGBoost weights when the user group characteristics are significant.

[0077] By using the Stacking ensemble model, it is possible to simultaneously cover multiple dimensions of patterns, such as time series, users, and resource relationships, thereby improving prediction accuracy and providing a more reliable decision-making basis for subsequent dynamic warm-up strategies.

[0078] S42: Transfer learning and online update mechanism. For special scenarios such as promotions, the model parameters finely tuned after training on historical promotion data are used as the initial values ​​for the new scenario. A fine-tuning strategy is used to accelerate the convergence speed of the model in the new scenario and reduce training time.

[0079] Deploy an online learning module to update the model hourly using the latest access data. Employ a partial update strategy to retain the core parameters of the model and only adjust the weights associated with the new data. This avoids catastrophic forgetting caused by full updates and ensures that the model can continuously adapt to changes in real-time access patterns.

[0080] S5: Dynamic preheating strategy generation. By selecting high-value resources, quantifying constraints, and optimizing scheduling, preheating efficiency is maximized while meeting system load and cache capacity limits. The specific implementation is as follows:

[0081] S51: High-probability resource screening. Based on the access probability ranking output by the prediction model, for example, resources with the highest access probability in the next 24 hours are prioritized for inclusion in the preheating list. At the same time, related resources are supplemented by combining the results of association rule mining. For example, when "mobile phone" is listed as a high-frequency access resource, highly related resources such as "mobile phone case" are automatically added to the preheating range to ensure comprehensive coverage of hot resources.

[0082] For new resources lacking historical access data, the cosine similarity between them and existing category resources can be calculated, with a threshold set to 0.8. This allows for the reuse of warm-up strategies from similar resources, thus solving the problem of warm-up decision-making during the cold start of new resources.

[0083] S52: Constraint Quantification and Optimization. A multi-objective optimization model is constructed, using resource access probability, resource size, node cost, and load coefficient as core parameters. A balancing factor coordinates the weights of each objective, forming an objective function that balances access value and system cost. A multi-objective genetic algorithm is used to solve the model, employing an elite retention strategy to select high-quality solutions and combining crowding calculations to maintain solution diversity. Finally, a Pareto optimal solution is generated, providing the optimal decision interval for the preheating strategy.

[0084] S53: Batch scheduling and load awareness. Based on the load prediction model built with XGBoost, it takes historical CPU utilization (5-minute granularity), remaining memory, network bandwidth, and other features as input, and outputs the system load prediction value for the next 4 hours.

[0085] Based on this, a dynamic batching algorithm is designed: when the predicted load is below 60%, 500 resources are loaded per batch; when the load is above 80%, the load is reduced to 200 resources per batch, adapting to the system's carrying capacity by flexibly adjusting the batch size. The batch interval adopts an exponential backoff strategy, with an initial interval set to 2 minutes. If a load conflict is detected, the interval time is automatically doubled to avoid the warm-up operation from impacting normal services.

[0086] S6: Preheating Execution and Adaptive Adjustment. Through a tiered loading strategy and real-time monitoring mechanism, preheating efficiency is ensured while avoiding excessive system stress, guaranteeing a stable and controllable preheating process. Specific implementation details are as follows:

[0087] S61: The hierarchical loading mode adopts a differentiated loading strategy based on the different access probabilities of resources.

[0088] For high-priority resources with an access probability of 80% or higher, a synchronous loading method is adopted, and the memory usage of the resources is reduced by 40% through LZ4 pre-compression technology to make efficient use of cache space.

[0089] For medium-priority resources with an access probability between 50% and 80%, an asynchronous loading method is adopted, using Redis pipelines for batch writing, and the token bucket algorithm is used to control the number of requests per second to within 2000, balancing loading speed and system load.

[0090] For low-priority resources with an access probability of less than 50%, the preheating operation will not be performed for the time being, but their access logs will be fully recorded to provide data support for subsequent model training and strategy optimization.

[0091] S62: Real-time monitoring and circuit breaker mechanism. Deploy a monitoring system consisting of Prometheus (an open-source monitoring and alerting system) and Grafana (an open-source monitoring dashboard tool) to collect key indicators such as cache hit rate, latency, and database query volume in real time, and present the system's operating status intuitively through a visual dashboard.

[0092] When the cache hit rate is detected to be below 70% for 5 consecutive minutes, the system automatically triggers the circuit breaker mechanism, suspends the current warm-up process, and switches to the LRU (Least Recently Used) caching strategy to prioritize the normal response of core business operations. After the system status returns to stability, the warm-up plan is re-evaluated and adjusted. This mechanism can effectively avoid system performance degradation caused by improper warm-up strategies and enhance the fault tolerance of the overall service.

[0093] S7: Effect Evaluation and Closed-Loop Optimization. Through multi-dimensional comparative analysis and dynamic parameter tuning, the effectiveness of the preheating strategy is continuously verified and iteratively upgraded to form a complete optimization closed loop. The specific implementation is as follows:

[0094] S71: Design a multi-dimensional comparative analysis control experiment. For example, assign 10% of users to the control group using the traditional LRU strategy, and include the remaining 90% of users in the experimental group using this method. Use a t-test to verify the difference in cache hit rates between the two groups, with a significance level set at p < 0.05 to ensure the statistical reliability of the experimental results. Simultaneously, calculate the relative increase in hit rate using the formula:

[0095] Accuracy improvement = (Experimental group accuracy - Control group accuracy) ÷ Control group accuracy × 100%

[0096] This quantifies the improvement in caching efficiency achieved by this method. In addition to hit rate, metrics such as system response time, database query volume, and cache space utilization are also monitored simultaneously to comprehensively evaluate the overall effect of the preheating strategy on system performance and resource allocation.

[0097] S72: Dynamic parameter tuning. For resources with excessive prediction errors, such as when the average error is greater than 15%, Bayesian optimization algorithms are used to automatically adjust model parameters, such as the learning rate of LSTM and the tree depth of XGBoost, to reduce prediction bias through iterative optimization.

[0098] Multiple policy templates are generated weekly based on historical data. A / B testing is used to compare the actual performance of different templates. When the difference between templates exceeds 20%, the best-performing template is selected as the new execution standard. This periodic iteration mechanism ensures that the cache hit rate remains above 95%, continuously improving the system's adaptability to high-concurrency scenarios.

[0099] This embodiment achieves intelligent cache preheating through data acquisition, model building, and strategy generation, thereby improving the preheating effect and system adaptability.

[0100] Example 2

[0101] This invention discloses an intelligent cache preheating system based on historical access data, comprising a data acquisition module, a preprocessing module, a pattern analysis module, a prediction model module, a strategy generation module, an execution adjustment module, and an evaluation and optimization module. The functions of each module and sub-module are as follows:

[0102] The data acquisition module is responsible for capturing and storing data accessed from all channels, comprising an omnichannel data access submodule and a distributed storage submodule. The omnichannel data access submodule deploys a Kafka streaming cluster, compatible with various transmission protocols, to handle user access requests from apps, mini-programs, web pages, and other channels. It balances data integrity and processing efficiency through a dynamic sampling mechanism and ensures data quality through multi-source data verification rules. The distributed storage submodule adopts a layered design. High-frequency access data is stored in HBase columnar format to support fast read and write operations, while historical full data is archived to HDFS and consistency is ensured through the RAFT protocol. Hash sharding and consistent hashing algorithms optimize the load balancing of storage nodes.

[0103] The preprocessing module transforms raw data into structured features suitable for model training, comprising a data cleaning submodule and a feature extraction submodule. The data cleaning submodule establishes a rule-based cleaning pipeline to standardize data formats, handle outliers and missing data, and identify and isolate abnormal traffic such as web crawlers. The feature extraction submodule constructs a four-dimensional feature matrix encompassing time, users, resources, and systems, vectorizing and reducing the dimensionality of unstructured data to form structured features.

[0104] The pattern analysis module mines access patterns to support the predictive model, and includes a time-series pattern recognition submodule, a user behavior clustering submodule, and an association rule mining submodule. The time-series pattern recognition submodule constructs an LSTM-ARIMA hybrid model, adapting to long-term trends and short-term fluctuations through a dynamic weight fusion strategy. The user behavior clustering submodule uses the K-means++ algorithm to group user feature vectors, monitoring behavior drift in real time and updating cluster centers. The association rule mining submodule uses the Apriori algorithm, incorporating a time decay factor and resource similarity screening rules to output high-value association rules.

[0105] The prediction model module improves the accuracy of predicting frequently accessed resources and includes an ensemble model training submodule and a dynamic update submodule. The ensemble model training submodule uses an attention mechanism to weightedly fuse features and constructs a Stacking ensemble model, fusing multi-dimensional prediction capabilities through base models (LSTM, XGBoost, LightGBM) and a meta-model (CatBoost). The dynamic update submodule employs transfer learning to accelerate model convergence for specific scenarios, deploying an online learning module to update the model hourly, and using partial update strategies to avoid catastrophic forgetting.

[0106] The strategy generation module generates efficient warm-up strategies, comprising a resource selection submodule, an optimization solution submodule, and a scheduling planning submodule. The resource selection submodule selects high-value resources based on access probability ranking, supplements related resources with association rules, and addresses the cold start problem for new resources through similarity calculation. The optimization solution submodule constructs a multi-objective optimization model and uses a multi-objective genetic algorithm to generate Pareto optimal solutions. The scheduling planning submodule designs a dynamic batching algorithm based on a load prediction model and adjusts the batch interval through an exponential backoff strategy.

[0107] The execution adjustment module ensures a stable and controllable preheating process, comprising a tiered loading submodule and a monitoring and circuit breaker submodule. The tiered loading submodule employs a differentiated loading strategy based on resource access probability: high-priority resources are loaded synchronously and pre-compressed; medium-priority resources are loaded asynchronously with controlled request volumes; and low-priority resources are not preheated but are logged. The monitoring and circuit breaker submodule deploys a Prometheus and Grafana monitoring system to collect key metrics in real time. When the cache hit rate is too low, the circuit breaker mechanism is triggered, pausing preheating and switching to an LRU strategy. Adjustments are reassessed and made after the system stabilizes.

[0108] The evaluation and optimization module continuously improves the preheating effect, including a comparative analysis submodule and a parameter tuning submodule. The comparative analysis submodule designs control experiments and evaluates the overall effect of the strategy through statistical tests and indicator monitoring. The parameter tuning submodule uses a Bayesian optimization algorithm to adjust the model parameters for resources with excessive prediction errors, generates strategy templates weekly, and iteratively optimizes them through A / B testing to maintain a high cache hit rate.

[0109] This system achieves intelligent and automated cache preheating through the collaborative work of its various modules, significantly improving system performance and stability.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A smart cache preheating method based on historical access data, characterized in that, Includes the following steps: S1: Uses a Kafka streaming cluster to access user access requests from all channels, and uses a two-tier storage architecture to store user access requests; S2: Standardize the format of access requests, handle outliers, and isolate traffic to construct a multi-dimensional feature matrix of "time-user-resource-system"; S3: Construct an LSTM-ARIMA model for temporal pattern recognition, cluster user behavior, and generate high-value association rules; S4: Construct an attention-weighted Stacking ensemble model, fuse multi-dimensional feature matrices through a hierarchical training mechanism, and optimize the model using online learning; S5: Generate a preheated resource list based on access probability ranking and association rules, construct a dynamic batching strategy using a multi-objective optimization algorithm, and adjust the batch interval through exponential backoff; S6: Implement a tiered loading mode and a monitoring circuit breaker mechanism, and combine optimization algorithms to fine-tune the parameters of resources for prediction errors.

2. The method according to claim 1, characterized in that, The dual-layer storage architecture includes HBase columnar storage for storing frequently accessed data and HDFS distributed storage for storing all data.

3. The method according to claim 1, characterized in that, In the LSTM-ARIMA model, the LSTM layer is set with 128 neurons to capture long-term trends, and the ARIMA parameters are determined through correlation analysis. A dynamic weight fusion strategy is used to adjust the output weights of LSTM and ARIMA.

4. The method according to claim 1, characterized in that, The clustering of user behavior includes using the K-means++ algorithm to group user feature vectors, prioritizing regions with higher data density when initializing centroids, and evaluating and determining the optimal number of clusters using silhouette coefficients.

5. The method according to claim 1, characterized in that, The process of generating high-value association rules involves using the Apriori algorithm, introducing a time decay factor, combining resource similarity calculated by the Jacobi index to screen effective rules, and establishing a multi-dimensional screening mechanism of support, confidence, and lift to generate high-value association rules.

6. The method according to claim 1, characterized in that, The Stacking ensemble model includes a base model composed of LSTM, XGBoost, and LightGBM, and a meta-model composed of CatBoost.

7. The method according to claim 1, characterized in that, The dynamic batching strategy includes, based on the XGBoost load prediction model, taking CPU utilization and memory remaining characteristics as inputs, outputting future load prediction values, and loading or reducing resources according to the load prediction values.

8. The method according to claim 1, characterized in that, Step S6 includes: S61: The hierarchical loading mode adopts a differentiated loading strategy based on the different access probabilities of resources; S62: Deploy a real-time monitoring system that triggers a circuit breaker mechanism when key indicators reach thresholds. These key indicators include cache hit rate, latency, and database query volume.

9. A smart cache preheating system based on historical access data, characterized in that, Includes the following modules: (1) Data acquisition module: responsible for accessing user access requests from all channels through the Kafka streaming processing cluster, using a two-layer storage architecture; (2) Preprocessing module: Performs cleaning operations such as format standardization, outlier handling, and traffic isolation on access requests, and extracts multi-dimensional features of time, user, resource, and system to construct a feature matrix; (3) Pattern analysis module: The LSTM-ARIMA hybrid model is used to mine time series patterns, the K-means++ algorithm is used to cluster user behavior, and the Apriori algorithm is used to generate association rules for high-value resources; (4) Prediction model module: Construct an attention-weighted Stacking ensemble model, combined with transfer learning and online update mechanism, to improve the prediction accuracy of high-frequency access resources in the future; (5) Strategy generation module: Based on access probability ranking and association rules, preheating resources are selected, and dynamic batching strategies are generated through multi-objective optimization algorithms. Combined with load prediction model and exponential backoff strategy, batch size and interval are adjusted to achieve efficient preheating scheduling. (6) Execution adjustment module: Implement a graded loading mode and ensure the stability and controllability of the preheating process through a monitoring system and circuit breaker mechanism; (7) Evaluation and optimization module: The preheating effect is evaluated through control experiments and multi-dimensional indicators, and the parameters are optimized and adjusted for resources with excessive prediction errors.

Images