Memory cache storage system oriented to big data flow

Abstract

The invention discloses a big data stream-oriented memory cache storage system, relates to the technical field of memory cache storage, and is used for solving the problem of poor memory cache storage management. According to the method, the access frequency, the timeliness and the service importance of the data are evaluated, the priority ranking model is constructed, the cache strategy is optimized based on the ranking result, the data storage and scheduling efficiency is improved, the write-in and read operation of the memory cache is adjusted in real time in combination with the flow prediction model, memory overload is effectively avoided, and the data storage and scheduling efficiency is improved. According to the method, non-emergency data writing is delayed by adopting a delayed writing and lazy loading strategy, data is loaded as required, meanwhile, a cache elimination mechanism is designed according to data priority, access frequency, survival time and size, data migration is carried out when a cache space is insufficient, reasonable resource allocation is ensured, and through a multi-level and dynamic cache management scheme, the cache management efficiency is improved. Therefore, the system performance, the resource utilization rate and the response speed are remarkably improved, the data processing efficiency is optimized, and cache conflicts are reduced.

Description

Technical Field

[0001] The present invention relates to the field of memory cache storage technology, and more specifically, to a memory cache storage system oriented to large data streams. Background Art

[0002] Memory cache storage for large data streams refers to the use of memory cache technology to efficiently store and access data when processing massive amounts of data to meet the needs of high throughput and low latency. Memory cache, as a temporary storage layer, can effectively alleviate the speed bottleneck of traditional disk storage and improve data processing efficiency. In the scenario of large data streams, due to the huge amount of data and high real-time requirements, memory cache storage provides a way to optimize the access performance of data streams by quickly accessing and dynamically managing cache content.

[0003] Deficiencies of existing technologies: In high-concurrency scenarios of large data streams, data is written very quickly, and a large amount of data is generated almost in real time. This data continuously enters the system, but since memory resources are limited, the cache system faces an ever-increasing amount of data traffic. As the amount of data increases, the cache update frequency is also increasing. This makes it difficult for traditional write mechanisms to schedule cache resources in a timely and accurate manner, especially in high-concurrency environments. Dynamic optimization of the cache becomes particularly difficult, and cache resources often cannot be reasonably allocated according to priority or timeliness, resulting in inefficient cache management. The system's response speed slows down when processing data, which can easily lead to cache overflow or a sharp drop in cache hit rate. This not only reduces the efficiency of data access, but also makes it impossible for data stream processing to maintain real-time and reliability. Summary of the invention

[0004] In order to overcome the above defects of the prior art, there is a solution as follows to solve the problem of unclear memory cache storage management in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The memory cache storage system for large data streams includes a memory optimization module, a traffic scheduling module, a delay loading module, and a strategy migration module, and each module is connected by signals;

[0007] The memory optimization module is used to obtain the individual priority of each piece of data based on the access frequency, timeliness, and business importance of the data, sort the data by priority, and adjust the cache strategy based on the sorting results;

[0008] Traffic scheduling module, which is used to build a traffic prediction model, predict data traffic changes in real time, adjust the data write and read operations of the memory cache according to the prediction results, and adjust the scheduling of data traffic with different priorities;

[0009] The lazy loading module is used to delay the writing and lazy loading of data, postpone the writing of non-urgent data, and load the content only when the data is requested;

[0010] The policy migration module is used to design a cache elimination mechanism based on data priority, access frequency, survival time, and size, and migrate data when cache space is insufficient.

[0011] In a preferred embodiment, the individual priority of each piece of data is obtained according to the access frequency, timeliness, and business importance of the data, and the data is prioritized and the cache strategy is adjusted according to the sorting result. The specific steps are as follows:

[0012] To evaluate the frequency of data access, set a time window W and record the number of data accesses N in the window. a (t): Among them, A i is the access status at time i, A i =1 means that the data is accessed at time i, A i =0 means that the data has not been accessed, and the number of accesses in the past time window period is calculated by sliding the window as the access frequency of the data;

[0013] Data timeliness evaluation is performed by obtaining the data generation time T data , current time T now , then the timeliness factor is expressed by the following formula: Among them, T max is the maximum validity time of the data;

[0014] Assess the business importance of data and calibrate it through business rules to indicate the impact of data on the system or application;

[0015] Compare the access frequency with the priority threshold to determine the access frequency priority of the data;

[0016] Compare the timeliness factor with the timeliness threshold to determine the timeliness priority of the data;

[0017] Compare the business importance weight with the business weight threshold to determine the business importance priority of the data;

[0018] By combining access frequency assessment, timeliness assessment, and business importance assessment, we can obtain the individual priority of each piece of data, and comprehensively rank the individual priorities of each data dimension to form the final data priority ranking.

[0019] In a preferred embodiment, the access frequency evaluation, timeliness evaluation and business importance evaluation are combined to obtain the individual priority of each piece of data, and the individual priorities of each data dimension are comprehensively ranked to form the final data priority ranking. The specific steps include:

[0020] The comprehensive priority P of each piece of data d total (d) is obtained by weighted summation, the formula is: P total (d) = w freq ·P freq (d)+w time ·P time (d)+w business ·P business (d), where w freq 、w time 、w business are the weight coefficients for access frequency, timeliness, and business importance, respectively. freq (d) is the access frequency priority of the data, P time (d) is the timeliness priority of the data, which is P business (d) business importance priority;

[0021] All data are sorted in descending order according to the comprehensive priority, and data are cached preferentially according to the sorting sequence number.

[0022] In a preferred embodiment, a traffic prediction model is constructed to predict data traffic changes in real time, and according to the prediction results, data write and read operations of the memory cache are adjusted to adjust the scheduling of data traffic of different priorities. The specific steps are as follows:

[0023] Based on historical data, the weighted moving average model is used as the traffic forecast model for prediction: in, is the flow at the predicted time t+1, R(ti) is the actual flow at the historical time ti, and w i is the weighting factor at time i, which is set according to the timeliness of the traffic;

[0024] According to the traffic prediction results, the traffic is divided into traffic increase and traffic decrease, and different traffic expiration control strategies are implemented;

[0025] The traffic expiration control strategy is: Among them, C alloc (t) is the cache space allocation at time t, α is the threshold of traffic surge, C max and C min They are the maximum cache space when traffic surges and the minimum cache space when traffic decreases;

[0026] The traffic expiration control strategy when traffic increases is to reserve cache space in advance and store high-priority data first;

[0027] The traffic expiration control strategy when traffic decreases is to release low-priority cache data, reduce memory usage, and retain critical business data.

[0028] In a preferred embodiment, a delayed writing and lazy loading strategy is used to postpone the writing operation of non-urgent data and load the content only when the data is requested, including the following steps:

[0029] Set the delayed write conditions and decide whether to postpone the write based on the importance of the data and the current cache resources. The trigger conditions for delayed write include data priority, system cache occupancy, and data write timing.

[0030] When the priority of the data is lower than the set priority threshold and the system cache occupancy rate of the data is greater than the preset cache occupancy threshold, the delayed write mechanism is enabled;

[0031] The delayed write data is stored in the buffer, and when the amount of delayed write data is greater than or equal to the batch threshold or the delayed write time is greater than the delay time threshold, the batch write operation is performed;

[0032] Set up the lazy loading trigger mechanism to trigger the lazy loading strategy.

[0033] In a preferred implementation, a lazy loading trigger mechanism is set to trigger the lazy loading strategy, and the specific steps are as follows:

[0034] Determine data access requests and system cache occupancy, and perform lazy loading on-demand loading and cache preloading based on the determined data access requests and system cache occupancy;

[0035] On-demand loading means that when data is requested, the system first checks whether the cache has stored the data. If the data is not in the cache, the system loads the data from the storage medium or external storage through the lazy loading strategy. If the data is already in the cache, the data is returned directly;

[0036] Cache preloading is performed in combination with traffic prediction. If the data is predicted to be frequently accessed in the future, the data is loaded into the cache in advance.

[0037] In a preferred embodiment, a cache elimination mechanism is designed according to the priority, access frequency, survival time and size of data, and data is migrated when the cache space is insufficient. The specific steps are as follows:

[0038] Determine the objects to be eliminated based on data priority, access frequency, survival time, and data size;

[0039] Dynamically select the elimination algorithm based on actual cache resource usage, data access patterns, and business needs;

[0040] Determine the cache migration strategy based on the memory usage threshold, access frequency, and cache space occupancy of the data.

[0041] The technical effects and advantages of the memory cache storage system for large data streams of the present invention are as follows:

[0042] The present invention constructs a data priority sorting model by comprehensively evaluating the access frequency, timeliness and business importance of data, and optimizes the cache strategy according to the sorting results to achieve efficient data storage and scheduling. The traffic prediction model predicts data traffic changes in real time, adjusts the write and read operations of the memory cache, and effectively avoids the problem of memory resource overload. In the cache management process, a delayed write and lazy loading strategy is adopted to postpone the write operation of non-urgent data and load the data only when it is requested, thereby optimizing memory usage and reducing unnecessary cache occupancy. In combination with the priority, access frequency, survival time and size of the data, a cache elimination mechanism is designed, and when the cache space is insufficient, space is released through data migration to ensure that system resources are reasonably configured. Through a multi-level and dynamic cache management strategy, the system performance and resource utilization are improved, data processing efficiency is optimized, cache conflicts and storage pressure are reduced, and the efficiency of memory resource use and system response speed are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a schematic diagram of the structure of a memory cache storage system for large data streams. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In order to achieve the above objectives, Figure 1 A schematic diagram of the structure of a memory cache storage system for large data streams of the present invention is given, which specifically includes a memory optimization module, a traffic scheduling module, a delay loading module and a strategy migration module, and each module is connected by a signal;

[0046] The memory optimization module is used to obtain the individual priority of each piece of data based on the access frequency, timeliness, and business importance of the data, sort the data by priority, and adjust the cache strategy based on the sorting results;

[0047] Traffic scheduling module, which is used to build a traffic prediction model, predict data traffic changes in real time, adjust the data write and read operations of the memory cache according to the prediction results, and adjust the scheduling of data traffic with different priorities;

[0048] The lazy loading module is used to delay the writing and lazy loading of data, postpone the writing of non-urgent data, and load the content only when the data is requested;

[0049] The policy migration module is used to design a cache elimination mechanism based on data priority, access frequency, survival time, and size, and migrate data when cache space is insufficient.

[0050] Step 1: Perform memory cache storage data priority evaluation and dynamic classification. According to the data access frequency, timeliness and business importance factors, evaluate the priority of each data and dynamically adjust the data classification, so as to achieve priority caching of high-priority data and delay or replacement of low-priority data, and maximize the resource utilization efficiency of the cache. The specific steps are as follows:

[0051] Evaluate the data access frequency. The data access frequency is an important dynamic parameter that directly affects the data priority. Usually, frequently accessed data is considered hot data and should be given a higher priority; infrequently accessed data is considered cold data and should be given a lower priority. The evaluation steps are as follows:

[0052] The access status of data is counted in real time through the sliding window mechanism. A time window W is set and the number of accesses N of data is recorded in the window. a (t): Among them, A i is the access status at time i, A i =1 means that the data is accessed at time i, A i =0 means the data has not been accessed;

[0053] The access frequency of data is dynamically updated by calculating the number of accesses within the past time window period through a sliding window;

[0054] Once the access frequency of data is calculated, the system can classify the data as hot data or cold data according to the access frequency. If the data is frequently accessed in a short period of time, it needs high priority caching; if the data is sparsely accessed, it can be assigned a lower priority. This dynamic adjustment mechanism can ensure that cache resources are concentrated on frequently accessed data and reduce storage waste of low-frequency data.

[0055] Access frequency can be used as the basis for data priority assessment, setting a priority threshold T f , when N a(t)>T f When N a (t)≤T f , the data is marked as low priority (cold data): Among them, P freq Priority after frequency of visit evaluation;

[0056] The timeliness of data is another important factor in determining its priority. Especially in real-time systems, the validity of data decreases over time. Data with high real-time requirements should be given higher priority and loaded into the cache first. For example, in real-time monitoring, financial transactions or IoT systems, the generation and validity of data have a time window. As time goes by, the value and validity of data will gradually decrease. Therefore, it is necessary to cache data with high timeliness requirements first and eliminate or downgrade data that is no longer valid in a timely manner.

[0057] The definition of timeliness parameter is to define a timeliness factor S(t) to indicate the validity of data since its creation. Let T data is the generation time of the data, and the current time is T now , then the timeliness factor can be expressed by the following formula: Among them, T max It is the maximum validity time of the data (such as the maximum validity period). The value of the validity factor S(t) is between [0, 1], indicating the validity of the data from creation to the current time;

[0058] Assign timeliness priority. The timeliness factor S(t) determines the timeliness priority of the data. Set a timeliness threshold T s To distinguish between high-timeliness data and low-timeliness data: Among them, P time The priority after timeliness evaluation, data with higher timeliness should be cached first.

[0059] In many scenarios, the business importance of data directly determines its priority. For example, transaction data or monitoring data is usually more important than ordinary log data, so it should be given a higher priority to ensure the location and access speed of monitoring data in the cache.

[0060] The business importance of data can be represented by a predefined weight value. A business weight function is set. For each piece of data d, its business importance weight can be calibrated according to business rules. It is usually an integer or floating value, indicating the degree of impact of the data on the system or application, for example, the integrity and accuracy of the data. Assume that the business importance weight of data d is W b(d), the business priority of the data can be expressed as: Among them, T b is the business weight threshold, which determines the allocation of high-importance data;

[0061] By combining the above three evaluations (access frequency, timeliness, and business importance), we can get the individual priority of each piece of data. In order to improve the flexibility and accuracy of data classification, we need to dynamically adjust and comprehensively sort the priorities of each data dimension to form the final data priority sorting, as follows:

[0062] The comprehensive priority P of each piece of data d total (d) can be obtained by weighted summation, the formula is: total (d) = w freq ·P freq (d)+w time ·P time (d)+w business ·P business (d), where w freq 、w time 、w business The weight coefficients are set for access frequency, timeliness, and business importance respectively.

[0063] All data are sorted according to the comprehensive priority, and the data with higher priority are placed in front and cached first.

[0064] Data priority evaluation and dynamic classification are core components of memory cache storage systems for large data streams. By comprehensively considering the access frequency, timeliness and business importance of data, it can dynamically evaluate the priority of data and ensure efficient storage of high-priority data in the cache through comprehensive priority sorting.

[0065] Step 2: Perform traffic prediction and intelligent traffic scheduling. In a memory cache storage system for large data streams, as data traffic fluctuates, the system needs to intelligently schedule the write and read operations of the memory cache to balance the storage requirements of high-priority data and the consumption of low-priority data. This not only depends on the accuracy of traffic prediction, but also requires real-time adjustment of cache strategies to avoid over-caching, cache conflicts, or unnecessary waste of storage resources. The specific steps of traffic prediction and intelligent traffic scheduling are as follows:

[0066] The data traffic in the big data stream is time-varying and uncertain. If the cache management system can predict the future data traffic changes, it will be able to schedule the traffic more reasonably and allocate cache space for high-priority data in advance to avoid problems such as insufficient cache resources or data overwriting. Therefore, a traffic prediction model is constructed to predict the data traffic within a certain period of time in the future.

[0067] The goal of traffic forecasting is to predict the traffic changes of specific data or data types in the future. Traffic forecasting not only considers short-term fluctuations, but also needs to combine long-term trends. The forecasting model mainly relies on the analysis of historical traffic data, including read frequency, write request rate, etc.

[0068] Select the prediction method, that is, select the appropriate prediction algorithm according to the characteristics of the data flow. This example uses the weighted moving average model based on historical data as the traffic prediction model for prediction, that is: in, is the flow at the predicted time t+1, R(ti) is the actual flow at the historical time ti, and w i It is the weighting factor at each moment, and the weight is usually set according to the timeliness of the traffic;

[0069] Historical traffic data: The monitoring system obtains past cache read and write traffic data, which can be divided into data at granularity such as minutes, hours, and days. These data can be used as training data for prediction models.

[0070] The traffic trend factor sets different weights according to the time window of historical data. For example, if the traffic fluctuations in the recent period are large, the weight of recent data will be increased. Usually, the data in the near distance (such as the traffic data in the past few seconds) has a greater impact on the prediction, so its weight is higher;

[0071] After each traffic prediction, the prediction error (e.g., the gap between the predicted value and the actual value) is calculated, and the accuracy of the model is evaluated based on the error so that the prediction model can be further optimized.

[0072] Design a traffic scheduling strategy. Based on traffic prediction, the traffic scheduling strategy determines how to dynamically allocate cache space according to the prediction results. A reasonable scheduling strategy can balance the storage requirements of high-priority data and low-priority data, optimize memory usage, reduce the cost of cache replacement, and improve the cache hit rate.

[0073] According to the traffic prediction results, the traffic is divided into traffic increase and traffic decrease, and different traffic expiration control strategies are implemented;

[0074] Based on the traffic forecast results, the goal of the real-time scheduling mechanism is to ensure the storage of high-priority data when traffic surges and reduce the cache burden when traffic decreases. When high traffic is predicted, the system needs to actively apply for more cache space, and when traffic decreases, the system can gradually release space;

[0075] Scheduling when traffic increases means that when the system detects a surge in traffic, it reserves enough cache space in advance based on the prediction results and gives priority to storing high-priority data. At this time, the cache management system will adopt priority scheduling (such as caching high-priority data in a high-priority location) and cache overflow strategies (such as data migration or data compression) to ensure that important data is not lost;

[0076] Scheduling when traffic decreases: If traffic is predicted to decrease, the system can release low-priority cache data or migrate some data to external storage to reduce memory usage and retain critical business data;

[0077] Traffic forecast value It can provide important decision-making basis for the cache management system. When the traffic surges, the system can pre-allocate a certain amount of cache space for the storage of high-priority data based on historical data analysis; when the traffic decreases, the system can reduce the use of cache space and free up memory for other tasks or applications. The specific method can be achieved through the following traffic expiration control strategy: Among them, C alloc (t) is the cache space allocation at time t, α is the threshold of traffic surge, C max and C min They are the maximum cache space when traffic surges and the minimum cache space when traffic decreases.

[0078] At the same time, dynamic resource adjustment based on cache pressure. In actual use, cache resource pressure may come from multiple aspects, including fluctuations in data traffic, cache space usage, current system load, etc. The system can adjust the cache scheduling strategy based on the current cache pressure. For example, when the cache pressure is high, the system can choose to compress or migrate some low-priority data to external storage and free up more space for high-priority data. By monitoring the current cache usage, the system cache pressure can be evaluated;

[0079] Traffic prediction and intelligent traffic scheduling are achieved through accurate traffic prediction and intelligent traffic scheduling. The system can achieve efficient cache resource management when facing uncertain and volatile data flows. Traffic prediction provides the ability to predict future data flows, while traffic scheduling uses this prediction information to flexibly allocate cache resources, thereby improving system performance and avoiding cache overflow or resource waste.

[0080] Step 3: Analyze the delayed writing and lazy loading strategies. By designing delayed writing and lazy loading strategies, the system can flexibly decide when to write and load data, thereby optimizing the use of memory resources, improving data processing efficiency, and ensuring timely access to high-priority data. The specific steps are as follows:

[0081] In the system, some data does not need to be written to the cache immediately. Instead, it can wait for a while until the cache has enough space or the data reaches a certain write threshold before performing the write operation. The core purpose of the delayed write strategy is to reduce the impact of high-frequency write operations on cache resources and improve the overall response performance of the system.

[0082] Delayed write conditions are set to decide whether to postpone writes based on the importance of the data and the current cache resources. In the memory cache system, a delayed write threshold needs to be set first, which is dynamically adjusted based on the system load, cache space, and data priority. Specifically, delayed write operations are allowed only when the system currently has sufficient cache resources and the data priority is low. The triggering conditions for delayed writes include:

[0083] Data priority P data , low-priority data can be written in a delayed manner, while high-priority data needs to be written into the cache in a timely manner;

[0084] System cache occupancy U cache ,If the cache resource occupancy exceeds the cache occupancy threshold γ, the system can decide to delay the write to avoid cache overflow;

[0085] Data writing timing T write ,Delayed writing needs to set the writing timing according to the current traffic and data generation time.,Delayed writing can be controlled by setting a timeout window, and writing is forced after the timeout;

[0086] The implementation of delayed writing is to store the data in the buffer first when the writing of data is delayed. The buffer is a temporary storage space that receives and caches the data to be written. The size, timeout policy and data cleaning mechanism of the buffer need to be reasonably set according to the traffic load. The longer the data is stored in the buffer, the higher the delay of the write operation, but it may also improve the cache utilization of the system;

[0087] A common strategy for delayed writing is batch writing. By accumulating a certain amount of data, the system can merge multiple data requests into a batch write operation. This method can reduce frequent write operations, reduce I / O overhead during writing, and improve the system's write efficiency.

[0088] In the delayed writing process, the cache occupancy rate is the cache occupancy ratio at the current time t, indicating the memory ratio of the system cache that has been used. If the cache occupancy rate is greater than the preset cache occupancy threshold, the delayed writing mechanism is enabled;

[0089] Set a delay time threshold, which indicates the maximum time that data is allowed to be delayed for writing. If the threshold is exceeded, the system will force writing. During the delayed writing process, data may enter multiple "to-be-written queues". Data with lower priority will enter the low-priority queue, and data with higher priority will be processed in advance.

[0090] The write batch threshold is to set a data batch threshold. When the amount of data to be written in the buffer reaches this threshold, the system will trigger a batch write operation;

[0091] The trigger condition for batch writing can be that the data accumulates to a certain amount (for example, N batch When the delay time exceeds the set threshold, it is forced to write;

[0092] When U cache >γ and P data When the data is lower than the set priority threshold, delayed writing is performed, and γ is the cache occupancy threshold; if the conditions are met, the data is stored in the buffer until the batch threshold B is reached. batch Or the delay writing time is greater than the delay time threshold T delay , and then perform a batch write operation;

[0093] The delayed write strategy is adjusted dynamically. As the system operating status changes, such as changes in cache resources and traffic fluctuations, the delayed write conditions also need to be adjusted accordingly. For example, when traffic increases and cache resources are tight, the system can lower the write threshold and reduce the triggering conditions for delayed writes; when traffic decreases and cache resources are sufficient, the system can relax the delayed write conditions and give priority to writing more data.

[0094] Lazy loading strategy aims to optimize the data loading and storage process and avoid unnecessary loading operations when the cache occupancy is too high. Lazy loading allows the system to load data into the cache only when it is actually needed, rather than loading the data immediately when it is generated, thereby reducing unnecessary cache access and loading operations, avoiding excessive memory usage, and ensuring timely response to high-priority data;

[0095] To set up the lazy loading trigger mechanism, first, data access request A data ,Only when data is requested, the system decides whether to load the data into the cache;

[0096] System cache occupancy U cache ,When the system's cache usage reaches a certain threshold, the lazy loading strategy will ensure that data is loaded only when necessary, thus preventing too much data from being loaded into the cache at the same time, causing excessive memory pressure;

[0097] The implementation method of lazy loading is set as follows:

[0098] 1. On-demand loading. When data is requested, the system first checks whether the cache has stored the data. If the data is not in the cache, the system will load the data from the storage medium or external storage through the lazy loading strategy. If the data already exists in the cache, the data is directly returned. This process delays the loading time of the data until the loading operation is actually performed when it is really needed.

[0099] 2. Cache preloading. To reduce latency, the lazy loading strategy can also be combined with traffic prediction for preloading. If it is predicted that certain data will be frequently accessed at some point in the future, the system can load the data into the cache in advance, but it will only be activated when the traffic prediction shows that the data is indeed needed.

[0100] The system should classify hot data and cold data based on historical access data. The lazy loading strategy should load cold data on demand, while hot data should be loaded first or use a preloading mechanism to ensure that hot data can be accessed and cached in a timely manner.

[0101] Step 4: Analyze the cache elimination and migration mechanism. Cache storage management needs to efficiently store and manage a large amount of data in a limited cache space. It is necessary to ensure that important data is retained first and reduce the waste of memory resources. The specific analysis steps are as follows:

[0102] The criteria for determining the objects to be eliminated should be based on a comprehensive judgment of multiple dimensions, such as data priority, access frequency, survival time, and data size. Different thresholds will be set for each factor, and only data that meets specific conditions will be marked as objects to be eliminated.

[0103] The priority of data reflects the importance of data in the current scenario. If the data has a lower priority, it is more likely to be eliminated. A priority threshold P is set. threshold , if P data If the value is less than the priority threshold, the data is considered to be eliminated first. data <P threshold ;

[0104] When the priority of data is low, it means that the data is not important for the current business or calculation, so it can be removed from the cache to free up cache space. In business scenarios, the importance of data usually depends on business logic and real-time requirements. For example, in sensor data, the historical data of some sensors may lose importance according to time, while real-time data still has a higher priority;

[0105] The access frequency of data determines whether the data is worth keeping in the cache. Data that has not been accessed for a long time should be eliminated first. Set an access frequency threshold Fashold , if the access frequency of data within a certain time window is F access (t) is less than the access frequency threshold, then the data is considered to be eliminated, that is, F access (t) <F ashold , access frequency F access The calculation formula of (t) in the time window Δt is: Among them, t i Indicates the timestamp of the i-th data access, is an indicator function, which takes the value 1 when the access occurs in the time interval [t-Δt, t], otherwise it takes the value 0;

[0106] The data survival time is the duration from the time the data enters the cache to the current time. Data with a longer survival time is usually more likely to no longer be used, so it can be eliminated. Similarly, a survival time threshold t is set laed , if the data survival time T age If the survival time threshold is exceeded, the data is considered to be eliminated, that is, T age >T ageshold , data survival time T age Calculated by the following formula: T age =t current -t laed , t current is the current timestamp, t laed is the last access time of the data;

[0107] The size of the data has a significant impact on the cache space occupied. Large data objects will occupy more cache space. Therefore, when cache resources are tight, larger data objects are eliminated first;

[0108] Set a data size threshold S threshold , if the size of the data is S data If the data size is larger than the data size threshold, it is considered that the data should be eliminated first, especially when the memory space is tight, that is, S data >S threshold ;

[0109] Taking all the above factors into consideration, the system can decide which data should be eliminated. In actual operation, a separate threshold for each condition is used for judgment. When all conditions are met, the data is considered a candidate for elimination.

[0110] In a cache management system, the selection of an elimination algorithm is the key to determining how to actually eliminate unnecessary data from candidate data. Different elimination algorithms have their own advantages and applicable scenarios in terms of performance, complexity, and resource consumption. Therefore, it is necessary to dynamically select the most suitable algorithm based on the characteristics of the data and the usage of cache resources.

[0111] In this system, due to the memory cache storage scenario for large data streams, the system needs to be able to select the most appropriate elimination algorithm based on the characteristics of the data (priority, access frequency, etc.) and the actual state of the cache (memory occupancy, access mode, etc.). The algorithms include:

[0112] LRU (least recently used), LFU (least frequently used), FI FO (first in, first out), adaptive hybrid algorithm (dynamic selection based on priority and resource status);

[0113] Before selecting an elimination algorithm, you need to first clarify the criteria for algorithm selection, that is, the frequency of data access. Frequently accessed data should be retained first.

[0114] Data priority: high-priority data should be retained even if it is not frequently accessed; cache space usage: when memory is tight, the system may need to quickly free up space; data time sensitivity: data with strong timeliness should be retained first if it is not updated in time;

[0115] Based on the above criteria, the system can dynamically select the appropriate elimination algorithm. The following are the specific steps;

[0116] Determine the algorithm based on the remaining space of the cache. If the free space of the cache is greater than the preset threshold, use a lightweight algorithm (such as FIFO or LRU). If the free space of the cache is less than the preset threshold, select an algorithm based on the priority and access frequency of the data.

[0117] According to the data priority selection algorithm, if the priority of the candidate data is low, LRU or LFU is used first. These algorithms are suitable for cold data elimination. If the priority of the candidate data is high, FIFO or a custom priority priority algorithm is used first to retain the high-priority data.

[0118] According to the access frequency selection algorithm, data with low access frequency can be eliminated using the LFU (least recently used) algorithm. LFU is suitable for data that has not been accessed for a long time. For data with high access frequency, the LRU (least recently used) algorithm can be used to eliminate data with a long access time;

[0119] According to the time-sensitivity selection algorithm, for data with strong timeliness (such as real-time monitoring data, alarm information, etc.), the FIFO or priority-first elimination strategy should be used first to ensure that older non-timeliness data is removed in a timely manner. For data with weak timeliness (such as historical records, log files, etc.), LRU or LFU can be used to ensure that frequently accessed data is retained;

[0120] Combining the above different elimination algorithms, the most appropriate algorithm can be dynamically selected according to the actual cache resource usage, data access pattern and business needs. For example, if the cache space is sufficient, the LRU or FIFO algorithm can be used to ensure that the data that has not been accessed for a long time or the oldest data is eliminated;

[0121] When cache space is tight, use the LFU algorithm or adaptive hybrid algorithm to dynamically determine the objects to be eliminated based on the access frequency and priority of the data;

[0122] When high priority data is used, ensure that high priority data (such as real-time data) is retained using FIFO or priority priority strategies.

[0123] This multi-dimensional dynamic selection algorithm approach ensures that cache space is utilized to the maximum extent while efficiently managing the cache, ensuring fast access to important data while reducing system resource consumption and latency.

[0124] Design a cache migration strategy. When cache space is insufficient, low-priority data can be migrated from memory to external storage to avoid wasting cache space and ensure that high-priority data is stored in a timely manner. By migrating unimportant data, the system can efficiently use memory and restore data as needed for subsequent access. Whether to migrate is determined by memory usage thresholds, access frequency, and cache space occupancy.

[0125] Data that has not been accessed for a long time and is infrequent is suitable for migration to external storage. When the access frequency is lower than the set threshold, the data can be marked as migration candidates, triggering data migration.

[0126] The definition of cache space occupancy is that when the cache space occupancy is close to the upper limit, data migration is triggered and the system needs to start the migration strategy. At this time, low-priority data should be migrated first to free up more space for high-priority data;

[0127] Memory usage threshold: When the cache space usage in the system exceeds the set threshold (for example, reaching 85% or 90%), data migration is triggered;

[0128] When the migration conditions are met, the next step is to select appropriate migration data from the current cache. To optimize the use of cache space, data can be selected based on the following criteria:

[0129] Combine the data priority and access frequency, and select the data with lower priority and lower access frequency as the migration candidate. Here, we can use the comprehensive sorting algorithm of priority and access frequency: migration score (x) = f(P data (x), F access(x)), where f is a decision function that can be selected according to the actual situation. Specifically, it can be designed as the following multidimensional decision function: Migration score Where ∈ is a constant to prevent division by zero errors;

[0130] For data migrated to external storage, if you need to access it later, you can quickly restore it through the following callback mechanism:

[0131] Lazy loading mechanism: When data is accessed, if the data has been migrated to external storage, the data is loaded from the external storage to the memory. At this time, the system should ensure that the delay in loading data from external storage does not affect the response time of the system;

[0132] Data preloading: For frequently accessed data, a preloading mechanism can be set up during the migration process. That is, when the data access volume increases, the data can be loaded back from the external storage to the memory in advance to reduce latency.

[0133] Through the design of the above steps, the cache migration strategy can dynamically select the data that needs to be migrated according to the real-time cache usage, data priority and access frequency, and ensure the availability of data through an efficient data recovery mechanism. The design of the cache migration strategy not only optimizes the utilization of cache space, but also improves the system's response speed and resource utilization efficiency, and adapts to data changes and needs in large data flow scenarios.

[0134] The threshold information in this embodiment is pre-set by professionals and will not be explained in detail here. Some parameter English letters in the embodiments have the same situation, but different meanings are explained when used, which will not be explained one by one here.

[0135] The present invention constructs a data priority sorting model by comprehensively evaluating the access frequency, timeliness and business importance of data, and optimizes the cache strategy according to the sorting results to achieve efficient data storage and scheduling. The traffic prediction model predicts data traffic changes in real time, adjusts the write and read operations of the memory cache, and effectively avoids the problem of memory resource overload. In the cache management process, a delayed write and lazy loading strategy is adopted to postpone the write operation of non-urgent data and load the data only when it is requested, thereby optimizing memory usage and reducing unnecessary cache occupancy. In combination with the priority, access frequency, survival time and size of the data, a cache elimination mechanism is designed, and when the cache space is insufficient, space is released through data migration to ensure that system resources are reasonably configured. Through a multi-level and dynamic cache management strategy, the system performance and resource utilization are improved, data processing efficiency is optimized, cache conflicts and storage pressure are reduced, and the efficiency of memory resource use and system response speed are significantly improved.

[0136] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0137] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0138] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0139] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0140] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0141] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A memory cache storage system for large data streams, characterized by: It includes memory optimization module, traffic scheduling module, delay loading module and strategy migration module, and each module is connected through signals; The memory optimization module is used to obtain the individual priority of each piece of data based on the access frequency, timeliness, and business importance of the data, sort the data by priority, and adjust the cache strategy based on the sorting results; Traffic scheduling module, which is used to build a traffic prediction model, predict data traffic changes in real time, adjust the data write and read operations of the memory cache according to the prediction results, and adjust the scheduling of data traffic with different priorities; The lazy loading module is used to delay the writing and lazy loading of data, postpone the writing of non-urgent data, and load the content only when the data is requested; The policy migration module is used to design a cache elimination mechanism based on data priority, access frequency, survival time, and size, and migrate data when cache space is insufficient.

2. The memory cache storage system for large data streams according to claim 1, characterized in that: It is used to obtain the individual priority of each piece of data based on the access frequency, timeliness, and business importance of the data, and to sort the data by priority. The cache strategy is adjusted according to the sorting results. The specific steps are as follows: To evaluate the frequency of data access, set a time window W and record the number of data accesses N in the window. a (t): Among them, A i is the access status at time i, A i =1 means that the data is accessed at time i, A i =0 means that the data has not been accessed, and the number of accesses in the past time window period is calculated by sliding the window as the access frequency of the data; Data timeliness evaluation is performed by obtaining the data generation time T data , current time T now , then the timeliness factor is expressed by the following formula: Among them, T max is the maximum validity time of the data; Assess the business importance of data and calibrate it through business rules to indicate the impact of data on the system or application; Compare the access frequency with the priority threshold to determine the access frequency priority of the data; Compare the timeliness factor with the timeliness threshold to determine the timeliness priority of the data; Compare the business importance weight with the business weight threshold to determine the business importance priority of the data; By combining access frequency assessment, timeliness assessment, and business importance assessment, we can obtain the individual priority of each piece of data, and comprehensively rank the individual priorities of each data dimension to form the final data priority ranking.

3. The memory cache storage system for large data streams according to claim 2, characterized in that: Combined with the access frequency assessment, timeliness assessment, and business importance assessment, the individual priority of each piece of data is obtained, and the individual priorities of each data dimension are comprehensively ranked to form the final data priority ranking. The specific steps include: The comprehensive priority P of each piece of data d total (d) is obtained by weighted summation, the formula is: P total (d) = w freq ·P freq (d)+w time ·P time (d)+w business ·P business (d), where w freq 、w time 、w business are the weight coefficients for access frequency, timeliness, and business importance, respectively. freq (d) is the access frequency priority of the data, P time (d) is the timeliness priority of the data, which is P business (d) business importance priority; All data are sorted in descending order according to the comprehensive priority, and data are cached preferentially according to the sorting sequence number.

4. The memory cache storage system for large data streams according to claim 3, characterized in that: It is used to build a traffic prediction model, predict data traffic changes in real time, adjust the data write and read operations of the memory cache according to the prediction results, and adjust the scheduling of data traffic with different priorities. The specific steps are as follows: Based on historical data, the weighted moving average model is used as the traffic forecast model for prediction: in, is the flow at the predicted time t+1, R(ti) is the actual flow at the historical time ti, and w i is the weighting factor at time i, which is set according to the timeliness of the traffic; According to the traffic prediction results, the traffic is divided into traffic increase and traffic decrease, and different traffic expiration control strategies are implemented; The traffic expiration control strategy is: Among them, C alloc (t) is the cache space allocation at time t, α is the threshold of traffic surge, C max and C min They are the maximum cache space when traffic surges and the minimum cache space when traffic decreases; The traffic expiration control strategy when traffic increases is to reserve cache space in advance and store high-priority data first; The traffic expiration control strategy when traffic decreases is to release low-priority cache data, reduce memory usage, and retain critical business data.

5. The memory cache storage system for large data streams according to claim 4, characterized in that: It is used to perform delayed writing and lazy loading strategies on data, postpone the writing of non-urgent data, and load content only when data is requested, including the following steps: Set the delayed write conditions and decide whether to postpone the write based on the importance of the data and the current cache resources. The trigger conditions for delayed write include data priority, system cache occupancy, and data write timing. When the priority of the data is lower than the set priority threshold and the system cache occupancy rate of the data is greater than the preset cache occupancy threshold, the delayed write mechanism is enabled; The delayed write data is stored in the buffer, and when the amount of delayed write data is greater than or equal to the batch threshold or the delayed write time is greater than the delay time threshold, the batch write operation is performed; Set up the lazy loading trigger mechanism to trigger the lazy loading strategy.

6. The memory cache storage system for large data streams according to claim 5, characterized in that: Set up the lazy loading trigger mechanism to trigger the lazy loading strategy. The specific steps are as follows: Determine data access requests and system cache occupancy, and perform lazy loading on-demand loading and cache preloading based on the determined data access requests and system cache occupancy; On-demand loading means that when data is requested, the system first checks whether the cache has stored the data. If the data is not in the cache, the system loads the data from the storage medium or external storage through the lazy loading strategy. If the data is already in the cache, the data is returned directly; Cache preloading is performed in combination with traffic prediction. If the data is predicted to be frequently accessed in the future, the data is loaded into the cache in advance.

7. The memory cache storage system for large data streams according to claim 6, characterized in that: It is used to design a cache elimination mechanism based on the data priority, access frequency, survival time, and size, and migrate data when the cache space is insufficient. The specific steps are as follows: Determine the objects to be eliminated based on data priority, access frequency, survival time, and data size; Dynamically select the elimination algorithm based on actual cache resource usage, data access patterns, and business needs; Determine the cache migration strategy based on the memory usage threshold, access frequency, and cache space occupancy of the data.

Images