Real-time big data stream processing method based on multi-level cache mechanism

Abstract

The invention discloses a real-time big data stream processing method based on a multi-level cache mechanism, and relates to the technical field of big data processing and real-time computing, in particular to the real-time big data stream processing method based on the multi-level cache mechanism, and a system comprises a data access layer, a multi-level cache unit, a stream processing computing engine and a result output module. The core of the multi-level cache unit is that the multi-level cache unit adopts a three-level cache architecture which at least comprises a memory, a solid-state storage and a disk, and is provided with a cache management scheduler which is used for dynamically scheduling data blocks according to data access frequency and retaining hotspot data in a high-speed cache. According to the method, the system receives data streams through the data access layer, and the data streams are stored in the multi-level cache after being pre-classified; and the stream processing calculation engine preferentially obtains data from the cache for real-time calculation. Through intelligent cache management and data pre-classification, the data access speed and the storage capacity are effectively balanced, and the real-time performance of big data stream processing and the system throughput are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of big data processing and real-time computing technology, specifically to a real-time big data stream processing method based on a multi-level caching mechanism. Background Technology

[0002] With the rapid development of the Internet of Things, mobile internet, and various smart applications, data is being generated at an unprecedented speed and scale, forming high-speed, continuous big data streams. How to process these data streams in real time with low latency and high throughput to quickly extract valuable information and make immediate responses has become a key challenge in the field of data processing.

[0003] Traditional stream processing systems typically rely on either memory or disk storage to process data streams. However, while relying solely on memory caching is fast, it is expensive and has limited capacity. When faced with sudden traffic spikes or continuous large-scale data surges, insufficient memory can easily lead to data loss or processing delays. On the other hand, relying solely on disk storage is insufficient to meet the stringent low-latency requirements of real-time processing due to the slow disk I / O speed.

[0004] While some existing technologies attempt to introduce caching mechanisms, most are only single-layer or simple two-layer caches, lacking sophisticated data lifecycle management and scheduling strategies. This prevents the system from dynamically optimizing based on the "hotness" (i.e., access frequency) of data; hot data may not reside in the cache, while cold data occupies valuable high-speed storage resources. Furthermore, the lack of intelligent pre-classification and prioritization when data flows into the system means that high-value, time-sensitive data may not receive priority processing, impacting overall processing efficiency and application value. Simultaneously, ensuring the temporal correctness of data processing results when the computing engine reads data in parallel from multi-level caches is also a significant technical challenge. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time big data stream processing method based on a multi-level caching mechanism. By setting up a clearing component and a cleaning component, it solves the problem mentioned in the background art that the depilation effect is not good when processing textile fabrics, and there are still hairs, which affect subsequent processing.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a real-time big data stream processing method based on a multi-level caching mechanism, wherein the system comprises: The data access layer is configured to receive continuously generated high-speed data streams from heterogeneous data sources through a high-throughput, low-latency communication mechanism, and to perform preliminary serialization and verification on the data streams. The multi-level cache unit is interconnected with the data access layer through an internal high-speed network. The multi-level cache unit includes at least a level-one memory cache based on dynamic random access memory, a level-two solid-state storage cache based on the non-volatile memory Express protocol, and a level-three disk cache based on a mechanical hard disk array, which are used for tiered temporary storage according to data lifecycle and access speed requirements. The stream processing computing engine, connected to the multi-level cache unit via memory mapping or direct memory access technology, is configured to proactively retrieve data from the multi-level cache unit according to the priority order of data residency level for real-time computing and analysis based on the urgency of the computing task. The result output module outputs the incremental processing results or aggregated statistical results generated by the stream processing computing engine through message push or database write interface.

[0007] Furthermore, the multi-level cache unit also includes a cache management scheduler, which serves as a standalone kernel-level service or firmware running on a dedicated coprocessor, and is used for: Continuously collect and monitor metrics such as space utilization of cache at all levels, access timestamps of data blocks, access frequency, and data size; Based on a preset, configurable data heat decay model and migration threshold strategy, asynchronous migration operations of data blocks are initiated between different levels of cache. Specifically, this includes promoting frequently accessed data blocks from low-speed cache to high-speed cache, and degrading data blocks with long retention times or access frequencies below a set threshold from high-speed cache to low-speed cache. The determination of data popularity takes into account a weighted calculation of recent access frequency and access time.

[0008] Furthermore, the first-level memory cache is optimized to cache the data fragments that are determined to be the most popular. Its internal data replacement strategy adopts an improved combined weight algorithm based on the principle of access timestamp proximity and access frequency factor. This algorithm maintains a popularity score for each data block and dynamically updates the score according to the latest access behavior, which serves as the basis for replacement.

[0009] Furthermore, the secondary solid-state storage cache is organized to cache data blocks with access frequency lower than the primary cache but significantly higher than the tertiary cache. Its internal data organization format adopts a column-oriented storage structure, storing attribute values ​​belonging to the same data record consecutively to optimize the batch scanning efficiency of specific data columns by the stream processing computing engine when performing range queries or aggregation calculations.

[0010] Furthermore, the data access layer also includes a data pre-classification module, which integrates a lightweight parsing rule engine to perform real-time parsing of the content characteristics of data packets in the early stage of data inflow, or to match them according to predefined business rules, thereby labeling the accessed data units with tags that represent processing priorities; based on the priority tags, the system initially distributes the data to the corresponding cache level in the multi-level cache unit or a specific priority queue within that level for buffering through built-in routing logic.

[0011] Furthermore, the stream processing computing engine is configured to have multi-path data reading capability, enabling it to simultaneously read different data blocks from the primary memory cache and the secondary solid-state storage cache in parallel for computation through independent I / O threads; the system includes a data consistency coordinator, which maintains a global logical clock or order.

[0012] Furthermore, the system also includes an asynchronous persistence controller, which runs in a background service thread pool and is responsible for asynchronously and sequentially writing the result data processed by the stream processing computing engine and the original data in the three-level disk cache that has reached the retention period to a long-term storage system such as a distributed file system or object storage in a batch group submission manner, so as to ensure data reliability and release cache space.

[0013] Furthermore, the method includes: The network interface component in the data access layer continuously receives and parses streaming data from external data sources. A multi-level cache unit data receiving queue is used to perform layered buffering of the continuously flowing data stream, where newly arrived data that is determined to be of high priority or unknown popularity first enters the first-level memory cache. The cache management scheduler periodically scans cache metadata, monitors data access patterns in real time, and migrates data blocks whose access frequency drops below a certain threshold from the high-speed cache level (such as the first-level memory cache) to the low-speed cache level (such as the second-level solid-state storage cache and the third-level disk cache) according to preset strategies. The stream processing computing engine prioritizes data from the first-level memory cache for real-time computing based on the priority set by the computing task itself and the location of the data in the current multi-level cache. When the required data is insufficient or does not exist in the first-level memory cache, it will send data retrieval requests to the second-level solid-state storage cache and the third-level disk cache in turn. The incremental results, alarm events, or statistical aggregation results generated by the calculation module are published or stored to external systems through the adapted interface in the result output module.

[0014] Furthermore, prior to the step of hierarchically buffering the data stream using multi-level cache units, a data preprocessing stage is also included: The incoming raw data byte stream is parsed in real time to extract its characteristic fields such as key values, timestamps, and event types; The extracted features are matched based on a preloaded rule base, or an initial priority identifier is assigned to the data through a lightweight computing model. Based on the initial priority identifier and the current load status of each level of cache, the routing decision logic determines the specific level and queue at which the data unit initially enters the multi-level cache unit.

[0015] Furthermore, the method also includes a cache warm-up phase before system startup or traffic peak: Based on the data access patterns analyzed from historical operation logs, or according to the explicit configuration by the operation and maintenance personnel, in the early stage of system startup or before the expected traffic peak, the identified historical high-frequency access data or key basic data on which the system operation depends are pre-loaded from the level 3 disk cache or long-term storage system to the faster level 2 solid-state storage cache, and even some core data is loaded into the level 1 memory cache, in order to improve the processing performance in the initial stage of the system.

[0016] This invention provides a real-time big data stream processing method based on a multi-level caching mechanism, which has the following beneficial effects: 1. By introducing a multi-level caching unit (including memory, solid-state storage, and disk cache), the system can dynamically adjust storage locations based on data access frequency. Frequently accessed data is always retained in the high-speed memory cache, allowing the stream processing engine to prioritize accessing this hot data, significantly reducing data retrieval latency. Simultaneously, the second-level cache uses a columnar storage format, optimizing the scanning efficiency of batch data and further accelerating the computation process. This hierarchical caching mechanism effectively avoids the bottleneck problems that may arise from a single storage layer, ensuring that the system can maintain low-latency processing even when high-speed data flows in, thereby significantly improving the processing efficiency of real-time data streams.

[0017] The cache management scheduler in the multi-level cache unit continuously monitors the space utilization and access patterns of each cache level and dynamically migrates data blocks based on preset strategies. For example, low-frequency data is automatically downgraded to a low-speed cache layer (such as disk), while high-frequency data is preferentially housed in a high-speed layer. This dynamic scheduling mechanism reduces the waste of expensive storage resources (such as memory) and extends hardware lifespan. Simultaneously, by assigning priority tags to data at the access layer through a data pre-classification module, the system can intelligently route high-value data directly to the high-speed cache, thereby reducing overall storage costs while ensuring the efficiency of critical data processing and achieving precise resource allocation.

[0018] The data pre-classification module in the data access layer can assign priority tags to data streams in real time based on data content characteristics or predefined rules, and route them to different cache levels. This design enables the system to flexibly handle sudden traffic surges or heterogeneous data sources (such as a mixture of high-priority business data and low-priority log data). Combined with the cache preheating function, the system can preload historical high-frequency data into the high-speed cache before traffic peaks, avoiding performance fluctuations caused by instantaneous pressure. This adaptive mechanism not only improves the system's stability in handling complex scenarios but also reduces reliance on manual intervention, making it suitable for industrial environments requiring long-term stable operation.

[0019] The stream processing engine supports parallel data reading from multi-level caches and ensures the temporal correctness of computation results through a data consistency coordinator. Even if data is scattered across different cache levels, the coordinator can synchronize processing progress, preventing analytical biases caused by disordered data reading order. Furthermore, the asynchronous persistence controller writes processing results and raw data in batches to the long-term storage system, avoiding the I / O pressure of real-time writes while ensuring data persistence without loss. This design balances the efficiency of real-time computing with a reliable foundation for data backtracking and auditing, making it particularly suitable for fields such as finance and surveillance where data accuracy is paramount. Attached Figure Description

[0020] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0021] Figure 1 This is a diagram illustrating the overall system architecture and data flow of the present invention. Figure 2 This is a flowchart of the data access and pre-classification process of the present invention; Figure 3 This is a flowchart illustrating the multi-level cache scheduling and management process of this invention. Figure 4 This is a flowchart of the data reading process of the stream processing computing engine of the present invention; Figure 5 This is a flowchart of the asynchronous persistence and cache preheating process of the present invention. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] How to use: I. Initial Configuration and Warm-up Phase 1. System Deployment and Parameter Setting: First, deploy each system component (data access layer, multi-level caching unit, stream processing computing engine, and result output module). Based on actual business needs, pre-configure key parameters, such as: Data access layer: Defines data source connection parameters, data parsing rules, and priority labeling rules (e.g., marking transaction data as "high" priority and log data as "medium" priority).

[0025] Multi-level cache units: Set capacity thresholds and data replacement strategies for each level of cache (memory, solid-state storage, disk). The data access layer continuously receives raw data streams from various data sources (such as message queues, sensor networks, and application logs).

[0026] The built-in data pre-classification module parses the data content in real time and assigns an initial priority label to each or each batch of data according to preset rules (such as data source, data type, and key field values).

[0027] Based on this priority label, data is initially routed to different levels of the multi-level cache unit or different queues within the same level. For example, "high" priority data can directly enter the fast channel queue of the first-level memory cache, while "medium" priority data can enter the second-level solid-state storage cache.

[0028] Tiered caching and dynamic scheduling: Newly arrived data is first stored in the first-level memory cache to ensure the fastest access speed.

[0029] The cache management scheduler continuously monitors the space utilization and data access frequency of each level of cache.

[0030] Based on monitoring results and preset strategies, the scheduler dynamically migrates data blocks between different levels of cache: Migrate data that has been accessed less frequently in the primary memory cache to the secondary solid-state storage cache.

[0031] Data that becomes cold in the secondary solid-state storage cache is migrated to the tertiary disk cache.

[0032] Conversely, when some data in the level 2 or level 3 cache is frequently accessed by the stream processing engine, the scheduler will promote it to a higher-speed cache level.

[0033] Real-time computing and processing: The stream processing engine proactively retrieves data from multi-level cache units based on the urgency of the computation task and data priority. It prioritizes retrieving data from the first-level memory cache.

[0034] To improve throughput, the stream processing computing engine can be configured to read data blocks in parallel from both the L1 memory cache and the L2 solid-state storage cache for computation.

[0035] The data consistency coordinator ensures that the timing of processing results is correct when data is read in parallel from different cache levels, avoiding logical errors caused by the time difference in the arrival of data at the computing engine.

[0036] When the data in the first-level cache cannot meet the computing needs, the engine will request data from the second-level and third-level caches in turn.

[0037] Results output and asynchronous persistence: The real-time processing results (such as aggregate statistics, anomaly alarms, and real-time reports) generated by the stream processing computing engine are immediately sent to the result output module for consumption by downstream systems (such as visualization platforms and alarm systems).

[0038] The asynchronous persistence controller runs in the background and is responsible for writing the processed result data and the raw data in the three-level disk cache to the long-term storage system (such as a data warehouse or distributed file system) in batches and efficiently for subsequent batch processing, historical queries or archiving.

[0039] III. Management, Monitoring and Optimization During use, operations and maintenance personnel should monitor key system indicators (such as cache hit rate, data access latency, computing engine processing latency, and queue backlog) through the management interface or interface provided by the system, and adjust parameters such as preheating strategy, priority rules, and cache replacement strategy according to the actual operating conditions to achieve continuous optimization of system performance.

[0040] Example: Example 1: Real-time Risk Monitoring System for Financial Transactions In the securities trading market, hundreds of thousands of orders and transaction records are generated every second, requiring millisecond-level risk control analysis to prevent illegal transactions (such as insider trading and market manipulation). Traditional batch processing methods have too high a latency and cannot meet the needs of real-time interception.

[0041] The application method of this system is as follows: First, at the data access layer, all market data and order flow data issued by exchanges are received in real time. The data pre-classification module assigns priorities to data based on characteristics such as security codes and transaction amounts, for example, marking large transactions and securities transactions on specific monitoring lists as "extremely high priority". This high-priority data is directly routed to the first-level memory cache. The cache management scheduler continuously monitors to ensure that these hot data always remain in the fastest memory layer. The stream processing computing engine is set to prioritize consuming data in the first-level cache, and it runs multiple risk control rule calculation models in parallel (such as abnormal transaction frequency detection and analysis of related account transaction behavior). Once the computing engine detects that a transaction has triggered a risk control rule, it immediately sends a risk alarm or interception instruction to the trading system through the result output module, thereby intervening in a very short time before or after the transaction is completed. At the same time, all raw transaction data and processing results are written to the historical database in batches by the asynchronous persistence controller for regulatory auditing. In this way, the system achieves real-time and accurate risk control of massive transaction data.

[0042] Example 2: Real-time Analysis and Early Warning of Sensor Data from a Large-Scale IoT Platform In smart factories or smart agriculture scenarios, tens of thousands of sensors (such as temperature, humidity, pressure, and vibration sensors) continuously generate monitoring data. These data streams need to be analyzed in real time to determine the operating status of equipment or environmental conditions and to detect anomalies promptly.

[0043] The usage method is as follows: The data access layer connects to a network of sensors distributed across various locations, receiving the time-series data reported by them. The pre-classification module can perform preliminary classification based on sensor type and preset thresholds. For example, vibration data from core production equipment is marked as "high priority," while ordinary environmental temperature and humidity data is marked as "medium priority." High-priority data enters the first-level memory cache, and medium-priority data enters the second-level solid-state storage cache (using columnar storage for easy batch scanning and calculation). The core task of the stream processing computing engine is to calculate the moving average and mutation rate of various indicators in real time and compare them with preset safety thresholds. The cache management scheduler dynamically adjusts the data location according to the access mode. For example, when a certain equipment area is under special attention, its related sensor data will be promoted to the high-speed cache. Once the engine analyzes that a sensor reading continuously exceeds the limit or undergoes a drastic change, it will immediately trigger an alarm through the result output module, notifying maintenance personnel to conduct on-site repairs. For regular data, the system performs aggregation calculations to generate trend reports. This method ensures a zero-delay response to anomalies in critical equipment while efficiently processing massive amounts of regular monitoring data.

[0044] Example 3: Real-time Personalized Recommendations and Anti-Fraud Measures on Large E-commerce Platforms During major e-commerce promotions, websites experience extremely high concurrent traffic, requiring real-time analysis of user behavior (clicks, browsing, adding to cart) and instant generation of personalized recommendation lists. Simultaneously, it's necessary to identify and block fraudulent activities such as fake orders.

[0045] The system operates as follows: All user behavior events (such as clicking on products and searching for keywords) are captured in real time by the data access layer. The data pre-classification module assigns priorities based on behavior type and user value (such as VIP users). For example, "add to cart" and "browse" behaviors of VIP users are set as high priority. This high-value behavior data is preferentially stored in the first-level memory cache to ensure the fastest processing. The stream processing computing engine performs two main tasks simultaneously: First, based on the user's behavior sequence in the last few minutes, it quickly reads data from the first and second-level caches and uses recommendation algorithms to update the recommended product list on the user side in real time; Second, it runs anti-fraud rules in parallel, such as detecting abnormally high-frequency ordering behavior from the same IP address. The calculation results are fed back to the user interface (updating recommendations) and the risk control center (marking suspicious orders) in real time through the result output module. The cache preheating strategy is particularly important in this scenario. Before the event starts, the system preloads information on best-selling products and core user profile data into the memory cache, greatly reducing the database pressure during peak periods.

[0046] Example 4: Real-time Power Fluctuation Balancing and Dispatch System for Energy Grids The State Grid Corporation of China needs to monitor the output power of power plants across the country and the load changes in electricity consumption areas in real time, and make millisecond-level adjustments to the grid frequency to maintain a dynamic balance between power generation and consumption and prevent grid collapse.

[0047] The system's application method for this critical task is as follows: The data access layer receives power data streams from thousands of power generation units and power consumption monitoring points. The pre-classification module marks all frequency-related critical balance signal data (such as output changes of large generator units and power flow of important lines) as "highest critical level," ensuring they enter the first-level memory cache with minimal latency. The stream processing computing engine processes this critical data with the highest priority, calculating the power deficit or surplus of the entire network in real time, and quickly calculating the capacity of standby units that need to be started or stopped, or the load that needs to be adjusted. The cache management scheduler ensures that the data required for these core calculations is always located in the fastest memory. The calculation results are instantly sent to power plants and load control centers through the result output module to execute frequency regulation commands. For non-real-time critical tasks such as historical load analysis and report generation, data from the second and third-level caches is used. The asynchronous persistent controller archives all operational data for post-event analysis and auditing. This system ensures the power grid has an ultra-fast response capability in the face of sudden power fluctuations.

[0048] Example 5: Real-time Optimization and Emergency Command of Urban Intelligent Traffic Flow To alleviate traffic congestion in large cities, traffic management departments need to process massive amounts of data in real time from intersection cameras, geomagnetic coils, GPS vehicles, etc., dynamically adjust traffic light timings, and plan the optimal routes for emergency vehicles (ambulances, fire trucks).

[0049] The specific usage process is as follows: Various types of traffic flow data are collected by the data access layer. The data pre-classification module marks data according to its urgency; for example, data identifying emergency vehicle passage and severe congestion on main roads are marked as "highest emergency priority," while traffic flow data at ordinary intersections is marked as "regular optimization priority." Emergency priority data is directly cached in the first-level memory cache. The stream processing computing engine prioritizes emergency data, calculates the optimal passage route for emergency vehicles in real time, and generates mandatory green light control instructions for traffic lights along the route. Simultaneously, the engine processes regular data in parallel, assesses regional traffic conditions in real time, and optimizes traffic light timing schemes. The cache management scheduler dynamically manages the data lifecycle; for example, after congestion on a certain road segment is relieved, the priority and cache position of its related data will be reduced accordingly. The result output module sends control instructions to the traffic light control system and publishes the traffic situation to the public. Through this method, the system achieves rapid response to sudden traffic events and continuous optimization of daily traffic flow.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time big data stream processing method based on a multi-level caching mechanism, characterized in that: The system includes: The data access layer is used to receive high-speed data streams from multiple data sources; A multi-level caching unit is connected to the data access layer. The multi-level caching unit includes at least a first-level memory cache, a second-level solid-state storage cache, and a third-level disk cache, which are used to perform layered temporary storage of data streams. A stream processing computing engine, connected to the multi-level cache unit, is used to retrieve data from the multi-level cache unit in priority order for real-time computing and analysis; And a result output module, used to output the processing results of the stream processing computing engine.

2. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 1, characterized in that: The multi-level cache unit further includes a cache management scheduler, which is used for: Monitor the space utilization and data access frequency of each level of cache; Data blocks are dynamically scheduled among different levels of cache according to a preset strategy; Among them, frequently accessed data is preferentially retained in the cache level.

3. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 2, characterized in that: The first-level memory cache is used to cache the most frequently accessed data shards, and its data replacement strategy is based on a weighted combination of access timestamps and access frequency.

4. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 2, characterized in that: The secondary solid-state storage cache is used to cache data blocks in secondary hotspots. Its data organization format is columnar storage to support the efficient scanning and calculation of the stream processing computing engine.

5. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 1, characterized in that: The data access layer also includes a data pre-classification module, which is used to assign priority tags to the access data streams according to the content characteristics of the data or predefined rules, and to initially route the data to different levels or queues of the multi-level cache unit based on the priority tags.

6. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 1, characterized in that: The stream processing computing engine is configured to simultaneously read data blocks from the first-level memory cache and the second-level solid-state storage cache for computation, and to ensure the timing correctness of the processing results through a data consistency coordinator.

7. The real-time big data stream processing method based on a multi-level caching mechanism according to claim 1, characterized in that: The system also includes an asynchronous persistent controller, which is used to write the result data processed by the stream processing computing engine and the original data in the three-level disk cache into the long-term storage system in batches.

8. A real-time big data stream processing method based on a multi-level caching mechanism, characterized in that, The method is applied to the system as described in any one of claims 1-7, and the method includes: Receives a continuous stream of input data through the data access layer; The data stream is buffered in layers using multi-level cache units, where newly arriving data first enters the first-level memory cache. The cache management scheduler monitors data access patterns in real time and migrates low-frequency access data from the high-speed cache level to the low-speed cache level; The stream processing computing engine prioritizes data from the first-level memory cache for real-time computation based on the priority of the computing tasks. When the data in the first-level memory cache is insufficient, it retrieves data from the second-level and third-level caches in turn. The results generated by the calculation are published through the results output module.

9. The method according to claim 8, characterized in that, Prior to the step of using multi-level cache units to perform hierarchical buffering of the data stream, the method further includes: Real-time analysis and feature extraction of the incoming data; Assign an initial priority to the data based on the extracted features or pre-configured rules; Based on the initial priority, the specific level and queue at which data initially enters the multi-level cache unit are determined.

10. The method according to claim 8, characterized in that, The method also includes a cache preheating phase: before system startup or in response to expected traffic peaks, historical high-frequency access data or critical basic data are preloaded from the level 3 disk cache to the level 2 solid-state storage cache or the level 1 memory cache.

Images