Heat-Aware Hybrid Memory Caching System and Method

Abstract

This invention provides a heat-aware hybrid memory caching system and method, comprising: unified abstraction and management of different media in hybrid memory; using a slab management mechanism combined with an allocation mechanism classified by size range to achieve hybrid memory allocation with different memory layers and different size distributions; marking the reference flag of migrated items through atomic operations and providing an asynchronous handle mechanism for external access during migration; placing cached item data in hybrid memory according to heat distribution with preset low-range overhead; using an LRU strategy to select eviction candidates for different media and different size classes in hybrid memory; and using a chained hash table to uniformly address all cached items in hybrid memory. This invention ensures that the caching system can correctly and efficiently utilize hybrid memory resources under different types of workloads, and solves the problem that slow media in hybrid memory have a significant impact on system performance.

Description

Technical Field

[0001] This invention relates to the field of hybrid memory caching technology, and more specifically, to a heat-aware hybrid memory caching system and method. Background Technology

[0002] Caching systems are widely used across all layers of modern web application architectures to reduce access to slower storage media and avoid redundant heavy computations. Typical application scenarios include geographically distributed static media data caching (CDN), application-side bypass caching, in-process temporary caching, caching for machine learning models (including input data caching and inference result caching), storage backend caching, and database caching. As can be seen, caching systems are widely used, appearing at all layers of large-scale system architectures in modern data centers. Therefore, in-depth analysis of the load characteristics and performance bottlenecks of caching systems, optimization of caching system performance, and reduction of caching system costs are of great significance.

[0003] Despite the diverse application scenarios of caching systems, they all face some common key challenges. One of the most prominent challenges is the massive working set of the cache, which refers to a group of objects with sufficient popularity for the system to benefit from caching them all. Therefore, to ensure a certain hit rate, a large working set requires a larger cache capacity. With the advent of the mobile internet and IoT era, the size of datasets is growing rapidly. On the one hand, while many application workloads approximate a Zipfian distribution, their popularity is not highly concentrated, exhibiting a significant long-tail effect, further increasing the size of the working set. On the other hand, the working sets of many application workloads change significantly over time, placing more stringent demands on the capacity and caching mechanisms of the caching system.

[0004] Traditional memory caching systems are increasingly struggling to meet the aforementioned key challenges. On the one hand, under the existing hardware architecture, the maximum DRAM capacity that a single server can support is limited and cannot be expanded arbitrarily as needed. On the other hand, DRAM is expensive, including both purchase costs and operating costs caused by its energy consumption. Under the current CPU and memory technology architecture, memory costs and energy consumption account for a considerable proportion of the total cost of a server, reaching 37.1% and 33.3% respectively, and are still showing an increasing trend with each generation.

[0005] Against this backdrop, recent work has attempted to increase the capacity of caching systems at a relatively low cost by using a hybrid approach of DRAM and SSD storage media, adapting to the ever-growing size of cache working sets. This type of work provides a transparent caching interface to upper-layer applications, maintaining the same byte-addressable granularity as existing memory caching systems, while hiding the complexity of the underlying hybrid storage media. Internally, it extends the capabilities of existing memory caching by implementing flash-based caching, managing the placement and eviction of cache objects of various sizes on flash media with pages as the smallest memory access unit.

[0006] However, these efforts still have some problems. First, there is an inherent contradiction between the large access granularity and write endurance of SSDs and the need to cache small-sized objects. In addition, the performance of SSDs is still far behind that of DRAM. For example, their read latency is three orders of magnitude lower than that of DRAM, and their read bandwidth is about 10 times lower. Therefore, the cache operation latency of such hybrid caching systems is significantly improved, but the throughput is reduced, and the performance instability is increased.

[0007] In recent years, the development of Storage Class Memory (SCM) has brought a new direction to solving the above problems. SCM has a larger capacity than traditional main memory DRAM and features byte addressability, low latency, and high bandwidth. It can be used as an extension of DRAM, expanding the capacity of DRAM with its lower price, larger capacity, and slightly lower performance.

[0008] The technique of adding other non-DRAM media as memory extensions in addition to DRAM is generally referred to as hybrid memory in academic research. The introduction of hybrid memory brings some new challenges. For example, the performance of the SCM (System-Driven Memory) in hybrid memory is relatively weak. Furthermore, considering the significant differences in the access frequency of different memory regions, the placement of data in hybrid memory has a crucial impact on overall system performance, requiring appropriate placement and migration strategies to optimize data placement within hybrid memory. Simultaneously, the additional overhead caused by these strategies also needs to be controlled.

[0009] Patent document CN115563235A discloses a hotspot-aware log structure merging tree read / write performance optimization method and related equipment, including: maintaining coarse-grained and fine-grained caches in memory, caching recently accessed data in the system at the granularity of data blocks and key-value pairs, respectively; the coarse-grained cache is a read-only cache used for service-wide query operations; the fine-grained cache is a read-write hybrid cache used for service-point query operations and write operations; a hotspot-aware data merging method is introduced on the hard disk, and a data heat calculation method is designed and implemented. Based on the data access heat, the execution of internal data merging operations in the log structure merging tree is scheduled, and invalid data participating in the merging is quickly pruned. After the merging is completed, the hottest new data blocks are prefetched into the coarse-grained cache.

[0010] Currently, some research work in other application scenarios is dedicated to solving similar problems, but there is a lack of research specifically on hybrid memory problems in caching system scenarios, including the design problems of caching systems under hybrid memory, and the placement and migration strategies of hybrid memory in caching system scenarios. Therefore, in order to make reasonable use of hybrid memory resources by the caching system, the following problems exist: (1) Cache items will be evenly distributed in media with different performance in hybrid memory, and the performance disadvantage of SCM will significantly affect the overall system performance. (2) The original eviction mechanism and addressing mechanism of the caching system cannot be directly adapted to the hybrid memory environment, which will affect the functionality of the caching system such as hit rate. (3) Potential migration operations will reduce concurrency scalability in order to ensure system consistency. Summary of the Invention

[0011] In view of the shortcomings of the prior art, the purpose of this invention is to provide a heat-aware hybrid memory caching system and method.

[0012] The heat-aware hybrid memory caching system provided by the present invention includes:

[0013] Hybrid Memory Management Module: Provides unified abstraction and management for DRAM and SCM media in hybrid memory. It uses the Slab management mechanism combined with an allocation mechanism classified by size range to perform hybrid memory allocation for different memory layers and different size distributions.

[0014] Concurrently Scalable Cross-Layer Migration Module: The reference flag of the migration item is marked through atomic operations, and an asynchronous handle mechanism is provided for external access during migration;

[0015] Placement migration strategy module: includes background migration strategy based on water level control, adjustable migration strategy based on probability control and migration strategy based on dynamic heat perception classification, so as to place cached item data in hybrid memory according to heat distribution with preset low range overhead;

[0016] Cache item management module: Uses the LRU strategy to select eviction candidates for different sizes of different media in the mixed memory, and uses a chained hash table to uniformly address all cache items in the mixed memory.

[0017] Preferably, the hybrid memory management module includes:

[0018] Hybrid Memory Resource Abstraction Management: Two types of memory resources are uniformly abstracted into a Segment interface, which represents a contiguous memory space with name and size attributes. The mapAddress function maps the memory to obtain a specific memory address, i.e., the starting position of the contiguous memory space. Then, depending on the memory medium, two implementations of this interface are provided: the ShmSegment class for DRAM memory resources and the DaxSegment class for SCM memory resources. The ShmSegment class contains the fd_ member variable, which is the file descriptor on the tmpfs file system used to establish the mapping. It implements the Segment interface by calling the shm_open and mmap system calls. The daxPath member variable in the DaxSegment class represents the path to the device descriptor corresponding to the SCM device, and the fd_ member variable represents the file descriptor corresponding to the opened device descriptor. It implements the Segment interface by calling the open and mmap system calls. Leveraging the DAX feature of the operating system, it directly accesses SCM memory resources through the mmap system call.

[0019] Slab management: Large contiguous memory resources are divided into coarse-grained blocks of 2MB each. Each 2MB memory block is a slab. Meta-information of each slab is maintained at the beginning of the large memory resource block, including a list of free slabs and a pointer to the location of newly allocated slabs. When allocating a new slab, if the list of free slabs is not empty, the free slabs in the list are reused directly; otherwise, the slab location pointer is moved forward and a new slab memory block is allocated.

[0020] AllocationClass Management: This relies on the AllocationClass class to allocate cache items of a specific size. Each AllocationClass instance is responsible for allocating cache items up to acSize. The number of AllocationClass instances and a series of acSizes are determined by setting a base size and a fixed ratio coefficient between two adjacent acSizes. During allocation, if the free alloc list is not empty, the free alloc in the list is reused directly; otherwise, the curr_offset pointer is moved forward, that is, a new alloc unit is allocated in the current slab block. If the pointer has reached the tail boundary of the slab block, it means that the slab has been exhausted, and a new slab block needs to be allocated.

[0021] Preferably, the cached item data includes:

[0022] Access container hook: Used to associate cached items with addressing access containers, enabling the system to find cached items through addressing access containers. The hook contains a compressed pointer to other cached items. The compressed pointer determines the location of the cached item through the memory layer, the slab number within the layer, and the cached item number within the slab.

[0023] Eviction container hooks: These are used to associate cache items with eviction containers, so that when the system selects an eviction object from the eviction container, the hook corresponds to the specific cache item.

[0024] Reference Flags: Used to maintain the reference count of cached items and related flags. The first 11 bits are reserved for other purposes, the middle 3 bits are used for internal reference counting, and the last 18 bits are used for external reference counting. This uint32 field is atomically updated using CAS. The internal reference count includes references to the cached item by the eviction container, references to the cached item by the access addressing container, and references to the cached item during the eviction and migration process. The external reference count represents the number of access rights to the cached item held above the user interface layer.

[0025] Expiration time: Used to clean up expired cached items;

[0026] Key-value length: Used to record the length of the key and value, where the first 8 bits are used for the length of the key and the last 24 bits are used for the length of the value;

[0027] Key-value array: The last part, which has a variable length, is used to store keys and values, and the length boundaries of both are determined by the key-value length field.

[0028] Preferably, the concurrent scalable cross-layer migration module includes: coordinating potential conflicts between migrations and between migrations and foreground access through atomic operations on reference flags; after checking the atomic flags, the foreground access operation only holds an asynchronous handle mechanism for the cache item being migrated, and the asynchronous handle mechanism held by it will be filled and awakened after the migration is completed. During the migration process, it is only necessary to lock the eviction container and the addressing access container separately in sequence and perform the actual migration operation between the two.

[0029] Preferably, the placement migration strategy module includes:

[0030] The background migration strategy based on water level control adopts a backend migration-primarily-migrated, foreground-triggered migration fallback design. Migration is triggered by a background thread at preset time intervals, reserving some free space for the high-performance memory layer. The execution intensity of migration is adaptively adjusted through a water level control mechanism. Above the high-performance memory layer, two threshold water levels are set according to the total capacity ratio: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The more the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the more the water level is below the upgrade water level, the stronger the upgrade migration should be. The candidates for background downgrade and upgrade migrations are obtained by traversing the tail and head of the corresponding eviction container LRU list, respectively.

[0031] Adjustable migration strategy based on probability control: On the basis of the main data flow path of the background migration strategy based on water level control, a new probabilistic branch flow path is introduced. At the corresponding decision node, the branch flow path is guided by a preset probability, while the rest remain on the main flow path. There are three branch flow paths: triggering upgrade migration when accessing cache items located in the SCM memory layer, placing new cache items directly in the SCM memory layer when inserting them, and directly evicting cache items freed up from the DRAM layer from the system without migrating to the SCM layer. The placement migration strategy is adjusted by adjusting the probability parameters of the three branch flow paths to adapt to different system configurations and actual load characteristics.

[0032] The migration strategy based on dynamic heat perception classification is as follows: a Count-Min-Sketch data structure is selected to estimate access frequency and reduce memory usage; an access frequency histogram data structure is maintained to reflect the overall distribution of cache item heat; the ratio of cold cache items to hot cache items is determined according to the capacity ratio of the system's DRAM memory layer and SCM memory layer, and then the access frequency threshold for distinguishing hot cache items is calculated; finally, the migration behavior is guided by the cold and hot cache item identification results, and objects with misplacement are migrated.

[0033] The heat-aware hybrid memory caching method provided by the present invention includes:

[0034] Hybrid memory management steps: Unify the abstraction and management of different media DRAM and SCM in hybrid memory, and use the Slab management mechanism combined with the allocation mechanism classified according to size range to perform hybrid memory allocation for different memory layers and different size distributions;

[0035] Concurrently scalable cross-layer migration steps: The reference flag of the migration item is marked through atomic operations, and an asynchronous handle mechanism is provided for external access during migration;

[0036] Placement migration strategy steps: Through a background migration strategy based on water level control, an adjustable migration strategy based on probability control, and a migration strategy based on dynamic heat perception classification, cached item data is placed in hybrid memory according to heat distribution with preset low range overhead.

[0037] Cache item management steps: Use the LRU strategy to select eviction candidates for different sizes of different media in the mixed memory, and use a chained hash table to uniformly address all cache items in the mixed memory.

[0038] Preferably, the hybrid memory management steps include:

[0039] Hybrid Memory Resource Abstraction Management: Two types of memory resources are uniformly abstracted into a Segment interface, which represents a contiguous memory space with name and size attributes. The mapAddress function maps the memory to obtain a specific memory address, i.e., the starting position of the contiguous memory space. Then, depending on the memory medium, two implementations of this interface are provided: the ShmSegment class for DRAM memory resources and the DaxSegment class for SCM memory resources. The ShmSegment class contains the fd_ member variable, which is the file descriptor on the tmpfs file system used to establish the mapping. It implements the Segment interface by calling the shm_open and mmap system calls. The daxPath member variable in the DaxSegment class represents the path to the device descriptor corresponding to the SCM device, and the fd_ member variable represents the file descriptor corresponding to the opened device descriptor. It implements the Segment interface by calling the open and mmap system calls. Leveraging the DAX feature of the operating system, it directly accesses SCM memory resources through the mmap system call.

[0040] Slab management: Large contiguous memory resources are divided into coarse-grained blocks of 2MB each. Each 2MB memory block is a slab. Meta-information of each slab is maintained at the beginning of the large memory resource block, including a list of free slabs and a pointer to the location of newly allocated slabs. When allocating a new slab, if the list of free slabs is not empty, the free slabs in the list are reused directly; otherwise, the slab location pointer is moved forward and a new slab memory block is allocated.

[0041] AllocationClass Management: This relies on the AllocationClass class to allocate cache items of a specific size. Each AllocationClass instance is responsible for allocating cache items up to acSize. The number of AllocationClass instances and a series of acSizes are determined by setting a base size and a fixed ratio coefficient between two adjacent acSizes. During allocation, if the free alloc list is not empty, the free alloc in the list is reused directly; otherwise, the curr_offset pointer is moved forward, that is, a new alloc unit is allocated in the current slab block. If the pointer has reached the tail boundary of the slab block, it means that the slab has been exhausted, and a new slab block needs to be allocated.

[0042] Preferably, the cached item data includes:

[0043] Access container hook: Used to associate cached items with addressing access containers, enabling the system to find cached items through addressing access containers. The hook contains a compressed pointer to other cached items. The compressed pointer determines the location of the cached item through the memory layer, the slab number within the layer, and the cached item number within the slab.

[0044] Eviction container hooks: These are used to associate cache items with eviction containers, so that when the system selects an eviction object from the eviction container, the hook corresponds to the specific cache item.

[0045] Reference Flags: Used to maintain the reference count of cached items and related flags. The first 11 bits are reserved for other purposes, the middle 3 bits are used for internal reference counting, and the last 18 bits are used for external reference counting. This uint32 field is atomically updated using CAS. The internal reference count includes references to the cached item by the eviction container, references to the cached item by the access addressing container, and references to the cached item during the eviction and migration process. The external reference count represents the number of access rights to the cached item held above the user interface layer.

[0046] Expiration time: Used to clean up expired cached items;

[0047] Key-value length: Used to record the length of the key and value, where the first 8 bits are used for the length of the key and the last 24 bits are used for the length of the value;

[0048] Key-value array: The last part, which has a variable length, is used to store keys and values, and the length boundaries of both are determined by the key-value length field.

[0049] Preferably, the concurrent scalable cross-layer migration step includes: coordinating potential conflicts between migrations and between migrations and foreground access through atomic operations on reference flags; after checking the atomic flags, the foreground access operation only holds an asynchronous handle mechanism for the cache item being migrated, and the asynchronous handle mechanism it holds will be filled and awakened after the migration is completed. During the migration process, it is only necessary to lock the eviction container and the addressing access container separately in sequence and perform the actual migration operation between the two.

[0050] Preferably, the placement and migration strategy steps include:

[0051] The background migration strategy based on water level control adopts a backend migration-primarily-migrated, foreground-triggered migration fallback design. Migration is triggered by a background thread at preset time intervals, reserving some free space for the high-performance memory layer. The execution intensity of migration is adaptively adjusted through a water level control mechanism. Above the high-performance memory layer, two threshold water levels are set according to the total capacity ratio: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The more the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the more the water level is below the upgrade water level, the stronger the upgrade migration should be. The candidates for background downgrade and upgrade migrations are obtained by traversing the tail and head of the corresponding eviction container LRU list, respectively.

[0052] Adjustable migration strategy based on probability control: On the basis of the main data flow path of the background migration strategy based on water level control, a new probabilistic branch flow path is introduced. At the corresponding decision node, the branch flow path is guided by a preset probability, while the rest remain on the main flow path. There are three branch flow paths: triggering upgrade migration when accessing cache items located in the SCM memory layer, placing new cache items directly in the SCM memory layer when inserting them, and directly evicting cache items freed up from the DRAM layer from the system without migrating to the SCM layer. The placement migration strategy is adjusted by adjusting the probability parameters of the three branch flow paths to adapt to different system configurations and actual load characteristics.

[0053] The migration strategy based on dynamic heat perception classification is as follows: a Count-Min-Sketch data structure is selected to estimate access frequency and reduce memory usage; an access frequency histogram data structure is maintained to reflect the overall distribution of cache item heat; the ratio of cold cache items to hot cache items is determined according to the capacity ratio of the system's DRAM memory layer and SCM memory layer, and then the access frequency threshold for distinguishing hot cache items is calculated; finally, the migration behavior is guided by the cold and hot cache item identification results, and objects with misplacement are migrated.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) By proposing a memory management and cache item management mechanism under hybrid memory, the problem of proper utilization of hybrid memory by the cache system is solved, and a unified and transparent cache capability can be provided to the outside world;

[0056] (2) By proposing a cross-layer placement migration strategy, the proportion of frequently used cache items placed in the DRAM layer is increased, thereby improving the overall system performance;

[0057] (3) By proposing a concurrent and scalable cross-layer migration mechanism, the concurrent contention overhead generated by cross-layer migration is reduced, thereby improving the concurrent scalability of the system. Attached Figure Description

[0058] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0059] Figure 1 This is a framework diagram of a heat-aware hybrid memory caching system;

[0060] Figure 2 A block diagram of a concurrent and scalable cross-layer migration mechanism;

[0061] Figure 3 This is a flowchart of a migration strategy based on heat distribution. Detailed Implementation

[0062] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0063] Example 1

[0064] This invention provides a heat-aware hybrid memory caching system, comprising:

[0065] Hybrid memory management section: It performs unified abstraction and management of different media DRAM (Dynamic Random Access Memory) and SCM (Storage Class Memory) in hybrid memory. It uses the Slab management mechanism combined with the allocation mechanism classified according to size range to perform hybrid memory allocation for different memory layers and different size distributions.

[0066] The concurrent scalable cross-layer migration section: Reference flags for migration items are marked using atomic operations, and asynchronous handles are provided for external access during migration, thereby reducing the scope of concurrency control and improving concurrency scalability.

[0067] The placement and migration strategy section includes three types of strategies: a water level-based background migration strategy, a probability-controlled adjustable migration strategy, and a migration strategy guided by dynamic heat-aware classification. These strategies are used to rationally place cached data in hybrid memory according to its heat distribution with low overhead.

[0068] Cache item management section: The LRU strategy is used to select eviction candidates for different sizes of different media in the mixed memory, and a chained hash table is used to uniformly address all cache items in the mixed memory.

[0069] The hybrid memory management component includes:

[0070] Hybrid Memory Resource Abstraction Management: Two types of memory resources are uniformly abstracted into a Segment interface, which represents a contiguous memory space with attributes such as name and size. The mapAddress function maps these resources to obtain a specific memory address, indicating the starting position of the contiguous memory space. Then, based on different memory media, two implementations of this interface are provided: the ShmSegment class for DRAM memory resources and the DaxSegment class for SCM memory resources. The ShmSegment class contains the fd_ member variable, which is the file descriptor on the tmpfs file system used to establish the mapping. It implements the Segment interface by calling the shm_open and mmap system calls. The DaxSegment class contains the daxPath member variable, which represents the path to the device descriptor corresponding to the SCM device, and the fd_ member variable, which represents the file descriptor corresponding to the opened device descriptor. It implements the Segment interface by calling the open and mmap system calls. Leveraging the DAX (Persistent Memory Direct Access) feature of the operating system, SCM memory resources can be directly accessed through the mmap system call.

[0071] Slab management: Large contiguous memory resources are divided into coarse-grained blocks of 2MB each, with each 2MB block constituting a slab. Meta-information for each slab is maintained at the beginning of the large memory block. The system maintains a list of free slabs and pointers to newly allocated slabs. When allocating a new slab, if the free slab list is not empty, a free slab in the list is reused; otherwise, the slab pointer is moved forward to allocate the new slab memory block.

[0072] AllocationClass Management: The system relies on the AllocationClass class to allocate cache items of a specific size. Each AllocationClass instance is responsible for allocating cache items up to acSize. The number of AllocationClass instances and a series of acSizes are determined by setting a base size and a fixed ratio coefficient between two adjacent acSizes. During allocation, if the free alloc list is not empty, the free allocs in the list are reused directly. Otherwise, the curr_offset pointer is moved forward, that is, a new alloc unit is allocated in the current slab block; if the pointer has reached the tail boundary of the slab block, it means that the slab is exhausted, and a new slab block needs to be allocated.

[0073] The cache item data structure includes:

[0074] Access container hooks: These are used to associate cached items with addressable access containers, enabling the system to locate cached items through the addressable access containers. Each hook contains a compressed pointer to another cached item. The compressed pointer determines the location of the cached item through the memory layer, the slab index within the layer, and the cached item index within the slab, eliminating the need for byte-level addressing and thus reducing pointer space usage.

[0075] Eviction container hooks: These are used to associate cached items with eviction containers, so that when the system selects an eviction object from the eviction container, it maps the hook to the specific cached item.

[0076] Reference Flags: Used to maintain the reference count of cached items and related flags. The first 11 bits are reserved for other purposes, the middle 3 bits are used for internal reference counting, and the last 18 bits are used for external reference counting. This uint32 field is atomically updated using CAS. The internal reference count includes references to the cached item by the eviction container, the access addressing container, and the eviction and migration processes; while the external reference count represents the number of access rights to the cached item held above the user interface layer.

[0077] Expiration time: Used to clean up expired cached items.

[0078] Key-value length: Used to record the length of the key and value. To save space, the first 8 bits are used for the length of the key, and the last 24 bits are used for the length of the value.

[0079] Key-value array: The last part, which has a variable length, is used to store keys and values, the length boundaries of which are determined by the key-value length field.

[0080] The concurrent scalable cross-layer migration mechanism addresses potential conflicts between migrations and between migrations and foreground accesses through atomic operations on reference flags. Foreground access operations, after checking the atomic flags, only hold an asynchronous handle for the cached item being migrated; this asynchronous handle is populated and awakened after the migration is complete. During migration, it is only necessary to lock the eviction container and the addressing access container separately and perform the actual migration operation between them, without needing to lock throughout the entire migration process.

[0081] The placement and migration strategy includes:

[0082] A water-level-controlled background migration strategy is adopted: To minimize interference with cache request execution in the system's foreground, a design is implemented where background migration is the primary method, with foreground-triggered migration as a fallback. Background threads trigger migrations at regular intervals, reserving a small amount of free space for the high-performance memory layer and preventing migrations from occurring on the critical path in the foreground. A water-level-controlled mechanism adaptively adjusts the migration execution intensity. Above the high-performance memory layer, two threshold water levels are set proportionally to the total capacity: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The greater the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the greater the water level falls below the upgrade water level, the stronger the upgrade migration should be. This fully utilizes the free space in the high-performance memory layer, optimizing the distribution of cached items. The candidates for background downgrade and upgrade migrations are obtained by traversing the tail and head of the corresponding eviction container's LRU list, respectively.

[0083] Probabilistic Placement and Migration Strategy Optimization: Based on the main data flow path of the above strategy, new probabilistic branch flow paths are introduced. At the corresponding decision nodes, the data flows are guided to the branch flow path with a certain probability, while the rest remain on the main flow path. There are three branch flow paths: triggering upgrade migration when accessing cache items located in the SCM memory layer; directly placing new cache items in the SCM memory layer when inserting them; and directly evicting cache items freed up from the DRAM layer from the system without migrating to the SCM layer. By adjusting the probability parameters of the three branch flow paths, the placement and migration strategy can be adjusted to adapt to different system configurations and actual load characteristics.

[0084] The migration strategy optimization based on dynamic heat perception classification involves using a Count-Min-Sketch data structure to estimate access frequency and reduce memory consumption. A histogram data structure of access frequency is maintained to reflect the overall distribution of cache item heat. The ratio of cold to hot cache items is determined based on the capacity ratio of the system's DRAM and SCM memory layers, and then the access frequency threshold for distinguishing hot cache items is calculated. Finally, the migration behavior is guided by the cold / hot cache item identification results, and objects with misplaced locations are migrated.

[0085] The cache item management section includes:

[0086] Cache eviction management: The LRU (Least Recently Used) strategy is used to select eviction candidates for different sizes of different media in mixed memory.

[0087] Cache item access addressing management: A chained hash table is used to perform unified addressing management for all cache items in the mixed memory.

[0088] Example 2

[0089] Example 2 is a preferred example of Example 1.

[0090] like Figure 1 This diagram presents the overall framework of the heat-aware hybrid memory caching system proposed in this invention. The lower half primarily showcases the memory management module for hybrid memory, including the utilization and abstraction methods of memory resources from different media, coarse-grained memory allocation in slab units, and a size-based allocation mechanism, effectively integrating hybrid memory resources and limiting memory fragmentation. The middle section's concurrent scalable cross-layer migration mechanism reduces concurrency overhead through asynchronous handles, atomic reference flags, and concurrent critical section reduction. The placement migration strategy section illustrates three strategies: a water-level controlled background migration strategy, a probability-adjusted placement migration strategy, and a dynamic heat-aware classification migration strategy. The upper part of the diagram illustrates the design of the system's cache item management, including the design of eviction containers and access-addressing containers under hybrid memory.

[0091] The design flow for the hybrid memory management section is described below. When a cache item of a specific size needs to be allocated on a certain memory layer, the MemAllocator instance on the corresponding memory layer is first selected, and then the smallest AllocationClass not smaller than the required size is found. Inside this AllocationClass instance, if the free alloc list is not empty, the free alloc in the list is directly reused. Otherwise, the curr_offset pointer is moved forward to allocate a new alloc unit in the current slab block; if the pointer has reached the tail boundary of the slab block, the SlabAllocator is called to allocate a new slab block. Inside the SlabAllocator, large contiguous memory resources are divided into coarse-grained blocks of 2M each, with each 2M memory block being a slab. Meta-information of each slab is maintained uniformly at the beginning of the large memory resource. When allocating a new slab, if the free slab list is not empty, the free slab in the list is directly reused; otherwise, the pointer is moved forward to allocate a new slab memory block. The aforementioned large contiguous memory resources are provided by the Segment interface, which provides a unified abstraction for the two types of memory resources. The two types of memory resources correspond to two implementations, ShmSegment and DaxSegment, respectively. The former maps to DRAM resources of the tmpfs file system through the shm_open and mmap system calls, while the latter maps directly to SCM resources through the open and mmap system calls and utilizes the DAX (Persistent Memory Direct Access) feature.

[0092] The design process for a concurrent, scalable, cross-layer migration mechanism is described below. For example... Figure 2As shown, the process begins by finding a cache entry in the current memory layer to free up space for the new allocation. An atomic operation is performed on its reference flag to mark it as being migrated. Then, the corresponding memory space is allocated in the next memory layer. When that memory layer is full, a cache entry to free up space is also needed. At this point, data copying between the cache entries in the two memory layers can begin. During the migration process, if the user performs a query operation on the cache entry, the original cache entry's location before migration is found in the addressing access container according to the original process. When adding an external reference to its reference flag, after determining that the cache entry is in a migration state, the system assembles an AsyncHandle object. This object does not point to a specific cache entry during creation; it only holds a WaitContext object. Access operations to the AsyncHandle will be blocked on its internal WaitContext. The semantics of this object are "cache entry hit, but data content not ready." Once the data is ready, the ready cache entry will be passed through the WaitContext, waking up the blocked user thread. After the migration is complete, a replacement operation will be performed in the addressable access container, replacing the old cache item with the new one, so that subsequent query operations can access the new cache item. In this design, the locking processes for all eviction containers and addressable access containers do not overlap. The critical path of locking only contains modification operations on the container itself and does not contain other locking or time-consuming operations, thereby improving the concurrency and scalability of the migration operation.

[0093] The design of the background migration strategy with water level control is described below. To reduce interference with the execution of cached requests in the system's foreground, a design is adopted where background migration is the primary method, with foreground-triggered migration as a fallback. Background threads trigger migration at regular intervals, reserving a small amount of free space for the high-performance memory layer and preventing the migration process from occurring on the critical path in the foreground. The execution intensity of migration is adaptively adjusted through a water level control mechanism. Above the high-performance memory layer, two threshold water levels are set according to the total capacity ratio: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The greater the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the greater the water level falls below the upgrade water level, the stronger the upgrade migration should be. This fully utilizes the free water level in the high-performance memory layer and optimizes the distribution of cached items. In summary, the intensity of the migration is determined by the distance between the actual water level in the high-performance memory layer and the threshold water level. When the threshold water level is exceeded, the corresponding migration will be temporarily interrupted. When the distance from the threshold water level is large, the upper limit of the migration intensity will be appropriately controlled, ultimately achieving a smoother migration intensity strategy that adapts to the load conditions.

[0094] The probabilistic placement migration strategy is described as follows: Based on the main data flow path of the above strategy, new probabilistic branch flow paths are introduced. At the corresponding decision nodes, the flow is guided to the branch flow path with a certain probability, while the rest remain on the main flow path. There are three branch flow paths. The first triggers upgrade migration when accessing cache items located in the SCM memory layer. Under equal probability, the likelihood of a more popular cache item being promoted is significantly higher than that of a less popular cache item, thereby increasing the proportion of popular cache items in the DRAM memory layer. The second places new cache items directly in the SCM memory layer when inserting them, avoiding the additional overhead caused by migrating a large number of cold cache items to the SCM layer after insertion. The third removes cache items freed up from the DRAM layer directly from the system without migrating them to the SCM layer, reducing the additional migration overhead caused by a large proportion of "one-hit twonders" in the load. By adjusting the probability parameters of the three branch flow paths, the placement migration strategy can be adjusted to adapt to different system configurations and actual load characteristics.

[0095] The design of the transfer strategy based on dynamic heat perception classification is described as follows: Figure 3 As shown, to perceive the popularity of cached items, the system needs to record the access frequency of cached items. To reduce memory usage, a Count-Min-Sketch data structure is chosen to estimate the access frequency. This is a two-dimensional array structure composed of counters, with each row corresponding to a different hash function. When recording an access operation for a cached item, a counter is determined in each row based on the hash calculation result of the cached item's key, and this counter is incremented. Correspondingly, when retrieving the access frequency of a cached item, the values ​​of the counters determined by the hash calculation results in each row are extracted, and the minimum value is an approximate estimate of its access frequency. To obtain the overall distribution of popularity, the system maintains an access frequency histogram data structure, as shown below. Figure 3 The middle section shows the range of access frequencies, where the horizontal axis represents the numerical range of access frequencies, and its numerical space is divided into several buckets. The vertical axis represents the number of cache entries whose access frequency falls within a certain bucket. Considering that most actual cache loads have obvious skewed characteristics, the system has made a special design for the range division of buckets, which will expand exponentially. The access frequency range of the nth bucket is [2]. n ,2 (n+1)This allows for the coverage of a wide range of access frequencies with a smaller number of buckets. When recording an access behavior of a cached item, the modified access count is recorded in the Count-Min Sketch structure. If the increased access count falls into a different bucket in the histogram than before the increase, the count of the original bucket is decremented by one, and the count of the new bucket is incremented by one. When classifying cached items as hot or cold, the ratio of cold to hot cached items is determined based on the capacity ratio of the system's DRAM and SCM layers. Then, the threshold for distinguishing hot cached items is determined by traversing the buckets. Figure 3 As shown in step ④, the system will periodically reduce the access frequency value of all cached items to reduce the impact of earlier access frequencies. Finally, the system will guide the migration behavior based on the hot / cold identification results of cached items. The objects that need to be migrated are those with "misplacement", namely, hot cached items placed in the capacity memory layer and cold cached items placed in the performance memory layer.

[0096] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0097] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A heat-aware hybrid memory caching system, characterized in that, include: Hybrid Memory Management Module: Provides unified abstraction and management for DRAM and SCM media in hybrid memory. It uses the Slab management mechanism combined with an allocation mechanism classified by size range to perform hybrid memory allocation for different memory layers and different size distributions. Concurrently Scalable Cross-Layer Migration Module: The reference flag of the migration item is marked through atomic operations, and an asynchronous handle mechanism is provided for external access during migration; Placement migration strategy module: includes background migration strategy based on water level control, adjustable migration strategy based on probability control and migration strategy based on dynamic heat perception classification, so as to place cached item data in hybrid memory according to heat distribution with preset low range overhead; Cache item management module: Uses LRU strategy to select eviction candidates for different sizes of different media in mixed memory, and uses a chained hash table to uniformly address all cache items in mixed memory; The hybrid memory management module includes: Hybrid Memory Resource Abstraction Management: Two types of memory resources are uniformly abstracted into a Segment interface, which represents a contiguous memory space with name and size attributes. The mapAddress function maps the memory to obtain a specific memory address, i.e., the starting position of the contiguous memory space. Then, depending on the memory medium, two implementations of this interface are provided: the ShmSegment class for DRAM memory resources and the DaxSegment class for SCM memory resources. The ShmSegment class contains the fd_ member variable, which is the file descriptor on the tmpfs file system used to establish the mapping. It implements the Segment interface by calling the shm_open and mmap system calls. The daxPath member variable in the DaxSegment class represents the path to the device descriptor corresponding to the SCM device, and the fd_ member variable represents the file descriptor corresponding to the opened device descriptor. It implements the Segment interface by calling the open and mmap system calls. Leveraging the DAX feature of the operating system, it directly accesses SCM memory resources through the mmap system call. Slab management: Large contiguous memory resources are divided into coarse-grained blocks of 2MB each. Each 2MB memory block is a slab. Meta-information of each slab is maintained at the beginning of the large memory resource block, including a list of free slabs and a pointer to the location of newly allocated slabs. When allocating a new slab, if the list of free slabs is not empty, the free slabs in the list are reused directly; otherwise, the slab location pointer is moved forward and a new slab memory block is allocated. AllocationClass Management: This relies on the AllocationClass class to allocate cache items of a specific size. Each AllocationClass instance is responsible for allocating cache items up to acSize. The number of AllocationClass instances and a series of acSizes are determined by setting a base size and a fixed ratio coefficient between two adjacent acSizes. During allocation, if the free alloc list is not empty, the free alloc in the list is reused directly; otherwise, the curr_offset pointer is moved forward, that is, a new alloc unit is allocated in the current slab block. If the pointer has reached the tail boundary of the slab block, it means that the slab has been exhausted, and a new slab block needs to be allocated.

2. The heat-aware hybrid memory caching system according to claim 1, characterized in that, The cached item data includes: Access container hook: Used to associate cached items with addressing access containers, enabling the system to find cached items through addressing access containers. The hook contains a compressed pointer to other cached items. The compressed pointer determines the location of the cached item through the memory layer, the slab number within the layer, and the cached item number within the slab. Eviction container hooks: These are used to associate cache items with eviction containers, so that when the system selects an eviction object from the eviction container, the hook corresponds to the specific cache item. Reference flags: Used to maintain the reference count of cached items and related flags. The first 11 bits are reserved for other purposes, the middle 3 bits are used for internal reference counting, and the last 18 bits are used for external reference counting. The uint32 field is atomically updated using CAS. The internal reference count includes references to the cached item by the eviction container, references to the cached item by the access addressing container, and references to the cached item during the eviction and migration process. The external reference count represents the number of access rights to the cached item held above the user interface layer. Expiration time: Used to clean up expired cached items; Key-value length: Used to record the length of the key and value, where the first 8 bits are used for the length of the key and the last 24 bits are used for the length of the value; Key-value array: The last part, which has a variable length, is used to store keys and values, and the length boundaries of both are determined by the key-value length field.

3. The heat-aware hybrid memory caching system according to claim 1, characterized in that, The concurrent scalable cross-layer migration module includes: coordinating potential conflicts between migrations and between migrations and foreground access through atomic operations on reference flags; after checking the atomic flags, the foreground access operation only holds an asynchronous handle mechanism for the cache item being migrated, and the asynchronous handle mechanism held by it will be filled and awakened after the migration is completed. During the migration process, it is only necessary to lock the eviction container and the addressing access container separately in sequence and perform the actual migration operation between the two.

4. The heat-aware hybrid memory caching system according to claim 1, characterized in that, The placement migration strategy module includes: The background migration strategy based on water level control adopts a backend migration-primarily-migrated, foreground-triggered migration fallback design. Migration is triggered by a background thread at preset time intervals, reserving some free space for the high-performance memory layer. The execution intensity of migration is adaptively adjusted through a water level control mechanism. Above the high-performance memory layer, two threshold water levels are set according to the total capacity ratio: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The more the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the more the water level is below the upgrade water level, the stronger the upgrade migration should be. The candidates for background downgrade and upgrade migrations are obtained by traversing the tail and head of the corresponding eviction container LRU list, respectively. Adjustable migration strategy based on probability control: On the basis of the main data flow path of the background migration strategy based on water level control, a new probabilistic branch flow path is introduced. At the corresponding decision node, the branch flow path is guided by a preset probability, while the rest remain on the main flow path. There are three branch flow paths: triggering upgrade migration when accessing cache items located in the SCM memory layer, placing new cache items directly in the SCM memory layer when inserting them, and directly evicting cache items freed up from the DRAM layer from the system without migrating to the SCM layer. The placement migration strategy is adjusted by adjusting the probability parameters of the three branch flow paths to adapt to different system configurations and actual load characteristics. The migration strategy based on dynamic heat perception classification is as follows: a Count-Min-Sketch data structure is selected to estimate access frequency and reduce memory usage; an access frequency histogram data structure is maintained to reflect the overall distribution of cache item heat; the ratio of cold cache items to hot cache items is determined according to the capacity ratio of the system's DRAM memory layer and SCM memory layer, and then the access frequency threshold for distinguishing hot cache items is calculated; finally, the migration behavior is guided by the cold and hot cache item identification results, and objects with misplacement are migrated.

5. A heat-aware hybrid memory caching method, characterized in that, include: Hybrid memory management steps: Unify the abstraction and management of different media DRAM and SCM in hybrid memory, and use the Slab management mechanism combined with the allocation mechanism classified according to size range to perform hybrid memory allocation for different memory layers and different size distributions; Concurrently scalable cross-layer migration steps: The reference flag of the migration item is marked through atomic operations, and an asynchronous handle mechanism is provided for external access during migration; Placement migration strategy steps: Through a background migration strategy based on water level control, an adjustable migration strategy based on probability control, and a migration strategy based on dynamic heat perception classification, cached item data is placed in hybrid memory according to heat distribution with preset low range overhead. Cache item management steps: Use the LRU strategy to select eviction candidates for different sizes of different media in the mixed memory, and use a chained hash table to uniformly address all cache items in the mixed memory; The hybrid memory management steps include: Hybrid Memory Resource Abstraction Management: Two types of memory resources are uniformly abstracted into a Segment interface, which represents a contiguous memory space with name and size attributes. The mapAddress function maps the memory to obtain a specific memory address, i.e., the starting position of the contiguous memory space. Then, depending on the memory medium, two implementations of this interface are provided: the ShmSegment class for DRAM memory resources and the DaxSegment class for SCM memory resources. The ShmSegment class contains the fd_ member variable, which is the file descriptor on the tmpfs file system used to establish the mapping. It implements the Segment interface by calling the shm_open and mmap system calls. The daxPath member variable in the DaxSegment class represents the path to the device descriptor corresponding to the SCM device, and the fd_ member variable represents the file descriptor corresponding to the opened device descriptor. It implements the Segment interface by calling the open and mmap system calls. Leveraging the DAX feature of the operating system, it directly accesses SCM memory resources through the mmap system call. Slab management: Large contiguous memory resources are divided into coarse-grained blocks of 2MB each. Each 2MB memory block is a slab. Meta-information of each slab is maintained at the beginning of the large memory resource block, including a list of free slabs and a pointer to the location of newly allocated slabs. When allocating a new slab, if the list of free slabs is not empty, the free slabs in the list are reused directly; otherwise, the slab location pointer is moved forward and a new slab memory block is allocated. AllocationClass Management: This relies on the AllocationClass class to allocate cache items of a specific size. Each AllocationClass instance is responsible for allocating cache items up to acSize. The number of AllocationClass instances and a series of acSizes are determined by setting a base size and a fixed ratio coefficient between two adjacent acSizes. During allocation, if the free alloc list is not empty, the free alloc in the list is reused directly; otherwise, the curr_offset pointer is moved forward, that is, a new alloc unit is allocated in the current slab block. If the pointer has reached the tail boundary of the slab block, it means that the slab has been exhausted, and a new slab block needs to be allocated.

6. The heat-aware hybrid memory caching method according to claim 5, characterized in that, The cached item data includes: Access container hook: Used to associate cached items with addressing access containers, enabling the system to find cached items through addressing access containers. The hook contains a compressed pointer to other cached items. The compressed pointer determines the location of the cached item through the memory layer, the slab number within the layer, and the cached item number within the slab. Eviction container hooks: These are used to associate cache items with eviction containers, so that when the system selects an eviction object from the eviction container, the hook corresponds to the specific cache item. Reference flags: Used to maintain the reference count of cached items and related flags. The first 11 bits are reserved for other purposes, the middle 3 bits are used for internal reference counting, and the last 18 bits are used for external reference counting. The uint32 field is atomically updated using CAS. The internal reference count includes references to the cached item by the eviction container, references to the cached item by the access addressing container, and references to the cached item during the eviction and migration process. The external reference count represents the number of access rights to the cached item held above the user interface layer. Expiration time: Used to clean up expired cached items; Key-value length: Used to record the length of the key and value, where the first 8 bits are used for the length of the key and the last 24 bits are used for the length of the value; Key-value array: The last part, which has a variable length, is used to store keys and values, and the length boundaries of both are determined by the key-value length field.

7. The heat-aware hybrid memory caching method according to claim 5, characterized in that, The concurrent scalable cross-layer migration steps include: coordinating potential conflicts between migrations and between migrations and foreground accesses through atomic operations on reference flags; after checking the atomic flags, the foreground access operation only holds an asynchronous handle mechanism for the cache item being migrated, and the asynchronous handle mechanism it holds will be filled and awakened after the migration is completed; during the migration process, it is only necessary to lock the eviction container and the addressing access container separately in sequence and perform the actual migration operation between the two.

8. The heat-aware hybrid memory caching method according to claim 5, characterized in that, The placement and migration strategy steps include: The background migration strategy based on water level control adopts a backend migration-primarily-migrated, foreground-triggered migration fallback design. Migration is triggered by a background thread at preset time intervals, reserving some free space for the high-performance memory layer. The execution intensity of migration is adaptively adjusted through a water level control mechanism. Above the high-performance memory layer, two threshold water levels are set according to the total capacity ratio: an upgrade water level and a downgrade water level, where the downgrade water level is lower than the upgrade water level. The more the water level exceeds the downgrade water level, the stronger the background downgrade migration should be; conversely, the more the water level is below the upgrade water level, the stronger the upgrade migration should be. The candidates for background downgrade and upgrade migrations are obtained by traversing the tail and head of the corresponding eviction container LRU list, respectively. Adjustable migration strategy based on probability control: On the basis of the main data flow path of the background migration strategy based on water level control, a new probabilistic branch flow path is introduced. At the corresponding decision node, the branch flow path is guided by a preset probability, while the rest remain on the main flow path. There are three branch flow paths: triggering upgrade migration when accessing cache items located in the SCM memory layer, placing new cache items directly in the SCM memory layer when inserting them, and directly evicting cache items freed up from the DRAM layer from the system without migrating to the SCM layer. The placement migration strategy is adjusted by adjusting the probability parameters of the three branch flow paths to adapt to different system configurations and actual load characteristics. The migration strategy based on dynamic heat perception classification is as follows: a Count-Min-Sketch data structure is selected to estimate access frequency and reduce memory usage; an access frequency histogram data structure is maintained to reflect the overall distribution of cache item heat; the ratio of cold cache items to hot cache items is determined according to the capacity ratio of the system's DRAM memory layer and SCM memory layer, and then the access frequency threshold for distinguishing hot cache items is calculated; finally, the migration behavior is guided by the cold and hot cache item identification results, and objects with misplacement are migrated.

Images