Weight-based key value storage cache elimination method and system and storage medium

Abstract

The invention discloses a weight-based key value storage cache elimination method and system and a storage medium, and the method comprises the steps: obtaining a plurality of candidate elimination data and a weight influence factor of each candidate elimination data in response to a cache elimination trigger signal; obtaining a weight value of each piece of candidate elimination data according to the weight influence factor of each piece of candidate elimination data; arranging the weight values of the candidate elimination data according to a sequence from small to large, and selecting the candidate elimination data ranked in the first n as final elimination data; and deleting the final eliminated data. According to the scheme, the weight values of the candidate elimination data are calculated according to the weight influence factors of the candidate elimination data such as the object size and the hit times, the n candidate elimination data with the lowest weight values are finally selected as the final elimination data, the hit rate is higher, meanwhile, the method can be suitable for the conditions of different storage object sizes, variable loads and the like, and the storage efficiency is improved. And the method is suitable for a KV cache system.

Description

technical field [0001] The invention relates to the design field of big data storage systems, in particular to a method, system and storage medium for weight-based key value storage and cache elimination. Background technique [0002] The cache elimination method is the core of the cache system and the determinant of the cache hit rate. At present, the commonly used cache elimination methods in the industry are based on LRU (Least Recently Used, least recently used), LFU (Least frequently used, least frequently used) and some combinations or local optimization algorithms derived from LRU / LFU. Currently, the KV (Key-Value, key-value storage) caching systems widely used in the industry mainly include Redis, Memcached and so on. Redis supports the LRU algorithm for random sampling, and also supports the LFU algorithm. The Memcached elimination strategy is based on LRU expiration and failure, which adopts the delay mode and is triggered when the key is accessed. Using a separ...

AI Technical Summary

Problems solved by technology

Using a separate cache elimination strategy, the problem is that it can only perform better in some specific workloads, as follows: (1) The LRU-based algorithm has a high hit rate in the hotspot access mode, but the sporadic , Periodic batch operations and scans will lead to a sharp drop in the LRU hit rate, and the cache pollution is more serious

(2) The algorithm based on LFU can effectively solve the problem of cache pollution caused by scanning, but LFU takes longer to adapt to the new access mode, and the calculation overhead is larger than that of LRU

(2) The workload is changeable, and the KV system may be used to cache various application data such as databases, pictures, and videos

[0004] (1) The current elimination algorithm assumes that the size range of the object is the same, which will reduce the KV cache hit rate; (2) cannot be applied to a variety of loads, and cannot be self-adaptive. When the workload changes , requires some manual parameter tuning to achieve a higher cache hit rate; (3) cannot support user-specified object priority, and cannot flexibly support some special needs of users

For example, in a caching system that supports multiple applications at the same time, users may need to cache some of the most important data first; (4) The data structures used by some algorithms require global synchronization, and the scalability is not high, and there are performance bottlenecks

Images