A non-uniform memory access aware key-value cache hierarchical scheduling method and system

Abstract

A hierarchical scheduling method and system for key-value caches aware of inconsistent memory access, relating to the field of large model inference technology. This application achieves reference counting and countdown mode switching by reusing a single byte metadata field. The key-value cache block status is updated when a request hits or ends its use. During eviction, a score is calculated for reference-counted blocks based on the normalized value of the most recent access timestamp, the affinity coefficient of the inconsistent memory access node, and the reference count. Blocks whose countdown has not reached zero are not included in the eviction candidate. Blocks with zero countdowns are scored based on access time and affinity coefficient, with high-scoring blocks prioritized for eviction and written to the next level of storage. This application solves the problems of existing technologies that fail to distinguish between the shared value of cache blocks and the eviction cost of inconsistent memory access, and are prone to intermittently eviction of high-value blocks due to requests. It improves cache hit rate with low metadata overhead, reduces cross-node recalculation, and increases system throughput.

Description

Technical Field

[0001] This application belongs to the field of large model inference technology, and in particular relates to a hierarchical scheduling method and system for non-consistent memory access-aware key-value cache. Background Technology

[0002] In large-model inference using a multi-GPU-based, non-consistent memory access architecture, paging mechanisms are commonly used to manage key-value caches to accelerate the decoding process. When cache space is insufficient, some key-value cache blocks need to be evicted. Existing eviction policies are mainly based on least recently used sorting, evicting all cache blocks in a uniform access time order, regardless of their degree of sharing or location on the storage node. Some schemes set a fixed proportion of reserved space for shared cache blocks to protect them from eviction within a certain area. However, the dependency levels of different shared cache blocks vary greatly, and a fixed reserved space cannot distinguish between high-value and low-value shared blocks. This could lead to low-value shared blocks occupying space for a long time or high-value shared blocks being mistakenly evicted outside the reserved space. Furthermore, the overhead of recalculating cache blocks across non-consistent memory access nodes after eviction is much higher than that of local node cache blocks, and existing eviction mechanisms do not incorporate this hardware cost difference into their decisions. In addition, references to shared cache blocks are often intermittent; their reference counts briefly drop to zero between requests. However, zeroing does not mean there are no subsequent references, and direct eviction would lead to a large amount of invalid recalculation, increasing decoding latency and system load. Summary of the Invention

[0003] This application provides a hierarchical scheduling method and system for non-consistent memory access-aware key-value cache. By reusing a single byte metadata field to dynamically switch between reference counting and countdown modes, integrating the normalized value of the most recently accessed timestamp, the affinity coefficient of non-consistent memory access nodes, and the reference count value into a comprehensive scoring calculation mechanism, and dynamically setting the initial value of the countdown buffer based on the average reference interval, this method accurately quantifies the shared value of cache blocks and distinguishes between local and cross-node eviction costs without increasing metadata overhead. This effectively avoids the false eviction of high-value shared blocks due to intermittent requests, ultimately reducing cross-node recalculation and improving cache hit rate and overall system throughput.

[0004] The first aspect of this application provides a hierarchical scheduling method for non-consistent memory access-aware key-value caches, including: When a request matches a key-value cache block, if the reuse metadata field indicates reference counting mode, the reference count of the key-value cache block is increased; if it indicates countdown mode, the reference count is switched to reference counting mode and the reference count value is set. When the request ends and the key-value cache block is no longer used, the reference count is decreased, and when the reference count decreases to zero, the countdown mode is switched to countdown mode and an initial countdown value is set based on the average reference interval. For key-value cache blocks in reference counting mode, the score is calculated based on the normalized value of the most recent access timestamp, the affinity coefficient of the non-consistent memory access node, and the reference count value; key-value cache blocks whose countdown value has not reached zero are not included in the eviction candidate; for key-value cache blocks whose countdown value has reached zero, the score is calculated based on the normalized value of their most recent access timestamp and the affinity coefficient. The key-value cache block with the highest score is evicted and written to the next level of storage; the reused metadata field is one byte, of which one bit is the mode bit and the remaining bits store the reference count value or countdown value; the affinity coefficient is determined based on the affinity relationship between the non-consistent memory access node where the key-value cache block is located and the current graphics processor.

[0005] Optionally, setting the countdown initial value based on the average reference interval includes: The number of erasure scan cycles experienced by key-value cache blocks in countdown mode within a set time window from entering the countdown to being re-referenced is calculated as the average reference interval, and the initial countdown value is set as the average value.

[0006] Optionally, when a request matches the key-value cache block, if the reuse metadata field indicates countdown mode, the request switches to reference counting mode and sets the reference count value while updating the most recently accessed timestamp of the key-value cache block.

[0007] Optionally, when increasing the reference count in reference counting mode, if the current reference count value reaches a preset upper limit, the reference count value remains unchanged.

[0008] Optionally, the calculation of the score based on the recent access timestamp normalized value, the affinity coefficient of the non-consistent memory access node, and the reference count value includes: The base score is determined based on the normalized value of the most recent access timestamp and the affinity coefficient. The base score is then adjusted in reverse using the reference count value, such that the higher the reference count value, the lower the score.

[0009] Optionally, the affinity coefficient is determined as follows: when the non-consistent memory access node where the key-value cache block is located belongs to the same node as the current graphics processor, the affinity coefficient is a first value; when they belong to different nodes, the affinity coefficient is a second value, and the first value is greater than the second value.

[0010] Optional, also includes: Each time a cache eviction is triggered, the countdown value of the key-value cache block that is in countdown mode and whose countdown value has not reached zero is decremented and updated.

[0011] Optionally, the key-value cache block with the highest eviction score further includes: For shared key-value cache blocks across nodes with inconsistent memory access, the remote graphics processor sharing the key-value cache block is notified via remote direct memory access to confirm that there are no references before the eviction process is performed.

[0012] Optionally, the notification to the graphics processor sharing the key-value cache block confirms that there is no reference includes: A query message is sent to the remote graphics processor that shares the key-value cache block. Only after receiving a no-reference confirmation from the remote graphics processor is the key-value cache block marked as ready for removal.

[0013] A second aspect of this application provides a non-consistent memory access-aware key-value cache hierarchical scheduling system, including at least one processor and a memory, wherein the memory stores instructions that, when executed by the processor, cause the system to perform the method described in the first aspect.

[0014] This application switches between reference counting and countdown modes by reusing metadata fields, enabling fine-grained recording of the dependency status of key-value cache blocks when a request hits or ends. When eviction is triggered and a score is calculated, for cache blocks in reference counting mode, the score is determined based on the normalized value of the most recent access timestamp, the affinity coefficient, and the reference count value: the higher the reference count, the lower the score is adjusted inversely. Cache blocks accessing non-consistent memory nodes also receive lower scores due to their lower affinity coefficients, thus placing them in a less likely position for eviction in the ranking. For cache blocks with a reference count of zero, they are not immediately included in the score calculation. Instead, an initial countdown value is set based on the average reference interval, putting them into countdown mode. During the countdown, they are not involved in eviction. The countdown decreases with each eviction scan. If a cache block is hit again during this period, it switches back to reference counting mode and updates the access timestamp. Only when the countdown ends does it calculate a score based on the normalized timestamp and affinity coefficient, entering the normal eviction candidate. Therefore, cache blocks with high dependency and cross-node access are prioritized for retention due to their lower scores. High-value cache blocks with intermittent no references are spared from false eviction during the buffer period, and after the buffer period, they compete for space with private cache blocks in a unified order. The entire process combines shared value quantification, awareness of the cost of eviction for inconsistent memory access, and prevention of intermittent false eviction. By reusing metadata fields and unifying them in the eviction score calculation, cross-node recalculation caused by false eviction can be reduced without increasing metadata overhead, thereby improving cache hit rate and system throughput, while not squeezing the space of private cache blocks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a non-consistent memory access-aware key-value cache hierarchical scheduling method in an embodiment of this application; Figure 2 This is a structural block diagram of a non-consistent memory access-aware key-value cache hierarchical scheduling system according to an embodiment of this application. Detailed Implementation

[0017] This application relates to the field of large model inference technology, and more particularly to a non-uniform memory access (NUMA)-aware key-value cache hierarchical scheduling method and system for large model inference. During large model inference, the autoregressive decoding-based generation method requires continuous storage of historical attention key-value pairs, i.e., a key-value cache (KV cache), to avoid redundant computation. The management efficiency of the key-value cache directly affects the system's throughput and inference latency.

[0018] In heterogeneous server architectures with multiple CPUs and GPUs, inconsistent memory access leads to significant differences in memory access latency between different nodes. Furthermore, key-value cache blocks may be shared by multiple inference requests, and the dependency and access frequency of different cache blocks vary. Existing cache eviction policies fail to adequately consider these factors, resulting in low cache hit rates and impacting system performance. This application optimizes the metadata management and eviction decision mechanism for cache blocks, enabling more accurate assessment of cache block value, reducing false evictions, and improving overall system performance.

[0019] like Figure 1 As shown, this application provides a hierarchical scheduling method for non-consistent memory access-aware key-value caches, including: When a request matches a key-value cache block, if the reuse metadata field indicates reference counting mode, the reference count of the key-value cache block is increased; if it indicates countdown mode, the reference count is switched to reference counting mode and the reference count value is set. When the request ends and the key-value cache block is no longer used, the reference count is decreased, and when the reference count decreases to zero, the countdown mode is switched to countdown mode and an initial countdown value is set based on the average reference interval. For key-value cache blocks in reference counting mode, the score is calculated based on the normalized value of the most recent access timestamp, the affinity coefficient of the non-consistent memory access node, and the reference count value; key-value cache blocks whose countdown value has not reached zero are not included in the eviction candidate; for key-value cache blocks whose countdown value has reached zero, the score is calculated based on the normalized value of their most recent access timestamp and the affinity coefficient. The key-value cache block with the highest score is evicted and written to the next level of storage; the reused metadata field is one byte, of which one bit is the mode bit and the remaining bits store the reference count value or countdown value; the affinity coefficient is determined based on the affinity relationship between the non-consistent memory access node where the key-value cache block is located and the current graphics processor.

[0020] In this embodiment, the metadata of the key-value cache block includes a reused metadata field. This field is used to record the dependency status and access information of the cache block. The reused metadata field is one byte long, with one bit serving as a mode bit to indicate whether the current cache block is in reference counting mode or countdown mode. The remaining seven bits are used to store the reference count value or countdown value. This design keeps the metadata overhead to a minimum and does not significantly impact the system's memory usage. When the mode bit is 1, it indicates that the cache block is in reference counting mode, and the remaining seven bits store the reference count value. When the mode bit is 0, it indicates that the cache block is in countdown mode, and the remaining seven bits store the countdown value. One byte is chosen as the length of the metadata field because one byte is the smallest addressable unit in a computer. Using one byte simplifies memory management and access operations, improving system efficiency. Using a smaller unit, such as a bit, would increase the complexity of memory access, requiring bitwise operations to obtain and modify the corresponding field values, which would actually reduce system performance. Meanwhile, the space of the on-chip static random access memory (SRAM) of the graphics processor is extremely limited. Excessive metadata overhead will consume a large amount of valuable on-chip resources, affecting the operation of other functional modules of the inference engine. Based on the calculation that a single card supports millions of key-value cache blocks, adding one byte of metadata to each block results in a total overhead of only about one megabyte, which is well within an acceptable range.

[0021] In this embodiment, when an inference request enters the system, a prefix matching operation is first performed. The system calculates the prefix hash value of the request input text and then searches the global key-value cache index table to see if a corresponding cache block exists. If a matching cache block exists, the key-value pairs in that cache block can be directly reused without recalculating the pre-filling. Prefix matching is a common optimization technique in large model inference, which can significantly reduce the computational overhead of the pre-filling stage, especially in scenarios with a large number of shared prefixes, such as fixed system prompts in customer service systems or historical context in multi-turn dialogues. The calculation of the prefix hash value can employ various hash algorithms, which can convert prefix text of different lengths into fixed-length hash values, facilitating fast lookup and matching.

[0022] In this embodiment, when a request matches a key-value cache block, the system first reads the reuse metadata field of the cache block and checks the status of the mode bit. If the mode bit indicates reference counting mode, the reference count value is incremented by 1. If the mode bit indicates countdown mode, the mode bit is switched to 1, entering reference counting mode, and the reference count value is set to 1. The increment operation of the reference count value is an atomic operation to avoid data race problems in the case of multi-threaded concurrent access. Atomic operation can ensure that only one thread can modify the reference count value at the same time, ensuring data consistency. In a multi-GPU system, cross-node reference count modification operations need to be completed through remote direct memory access to ensure data synchronization. Remote direct memory access can directly modify the data in the video memory of a remote GPU without the participation of the central processing unit, and has low latency and high efficiency.

[0023] It's important to note that the key-value cache block matched by the request may reside in the local graphics processor's memory or in the memory of another graphics processor within the same non-consistent memory access group. Both local and cross-node shared cache blocks follow the same reference counting increment / decrement rules. The pre-filling and recalculation overhead for cross-node shared cache blocks is significantly higher than for local cache blocks. Therefore, incorporating cross-node shared cache blocks into a unified reference counting management system accurately quantifies their shared value and prevents them from being easily evicted. In actual operation, the system maintains a global cache block location mapping table, recording the graphics processor number and memory address of each cache block, enabling rapid location and access when a cache block is matched.

[0024] In this embodiment, when a reasoning request finishes using a key-value cache block, the system decrements the reference count of that cache block by 1. There are two scenarios where a request finishes using the cache block: either the request completes reasoning normally and generates an end-of-time token, or the request times out and is forcibly terminated by the system. Covering these two scenarios avoids reference count leakage caused by abnormal request exits. Reference count leakage prevents the cache block's reference count from ever reaching zero, thus ensuring it is never evicted and eventually exhausting the cache space. The reference count decrement operation is also atomic to guarantee data consistency. In scenarios where cache blocks are shared across nodes, when a remote graphics processor finishes using the cache block, the reference count of that cache block needs to be modified via remote direct memory access to ensure that all graphics processors sharing the cache block see the latest reference count.

[0025] In this embodiment, when the reference count drops to zero, the system switches the mode bit of the cache block to 0, enters countdown mode, and sets the initial countdown value based on the average reference interval. A reference count of zero only indicates that no request is currently using the cache block; it does not mean that the cache block will not be referenced by other requests later. In many business scenarios, cache block references are intermittent. For example, during gaps in batch requests or off-peak periods, the reference count of a cache block may briefly drop to zero, but it will be referenced again by new requests after a period of time. If the cache block is evicted immediately after the reference count reaches zero, subsequent requests will need to recalculate pre-filling, increasing system overhead. Therefore, setting a countdown buffer period, allowing cache blocks with zero reference counts to remain unevicted during the buffer period, can effectively reduce false evictions caused by such intermittent references. The length of the countdown buffer period is dynamically determined by the average reference interval, adapting to different workload characteristics.

[0026] In this embodiment, the system triggers a cache eviction operation when the cache space utilization reaches a preset threshold. The preset threshold is typically set between 80% and 95%, and the specific value can be adjusted based on the actual system operation. Setting the threshold too high will cause the system to frequently trigger cache eviction operations, increasing system overhead. Setting the threshold too low will result in low cache space utilization, wasting hardware resources. The triggering of the cache eviction operation is only related to the cache space utilization; there is no timed scanning mechanism, thus avoiding unnecessary system overhead. The system monitors the cache space usage in real time, and immediately initiates the eviction scanning process when the utilization reaches the preset threshold.

[0027] In this embodiment, when making eviction decisions, the system first classifies all key-value cache blocks. For cache blocks in reference counting mode, a score is calculated based on the normalized value of the most recent access timestamp, the affinity coefficient of the non-consistent memory access node, and the reference count. Cache blocks in countdown mode with countdown values ​​not yet zero are not included in the eviction candidate. For cache blocks in countdown mode with countdown values ​​already zero, a score is calculated based on their normalized value of the most recent access timestamp and the affinity coefficient. Excluding cache blocks with countdown values ​​not yet zero from the eviction candidate provides a buffer period for cache blocks with zero reference counts, avoiding false evictions due to intermittent references. During the buffer period, if the cache block is re-referenced, it will switch back to reference counting mode and remain in the cache. If it is still not referenced after the buffer period ends, it indicates that the cache block has low value and can be evicted. This classification method can minimize false evictions while ensuring that cache space can be released in a timely manner.

[0028] In this embodiment, the most recently accessed timestamp is used to record the time when a cache block was last accessed. The precision of the most recently accessed timestamp can be adjusted according to the actual needs of the system, typically using millisecond or microsecond precision. Higher precision can more accurately reflect the access time order of cache blocks, but it will also increase the storage overhead of the timestamp. In practical applications, millisecond precision is usually sufficient for most scenarios. The calculation method of the normalized value of the most recently accessed timestamp is as follows: The system records the most recently accessed timestamps of all cache blocks and finds the maximum and minimum values. For each cache block, the minimum value is subtracted from its most recently accessed timestamp, and then divided by the difference between the maximum and minimum values ​​to obtain the normalized value. The normalized value ranges from 0 to 100; the larger the value, the later the most recently accessed time of the cache block. The purpose of normalization is to convert timestamps from different time ranges into a uniform value between 0 and 100, facilitating comparison and calculation. Without normalization, timestamps from different time ranges may differ significantly, leading to deviations in score calculation. For example, when the system runs for a long time, the timestamp value will be large, while the timestamp value of newly created cache blocks will be small. This will put newly created cache blocks at a disadvantage in score calculation, even if they are recently accessed.

[0029] In this embodiment, the affinity coefficient is determined based on the affinity between the non-consistent memory access node where the key-value cache block resides and the current graphics processor. When the node containing the key-value cache block and the current graphics processor belong to the same node, the affinity coefficient is a first value. When they belong to different nodes, the affinity coefficient is a second value, and the first value is greater than the second value. This design results in cross-node cache blocks receiving lower scores in the score calculation, making them less likely to be evicted. Because the overhead of recalculating cross-node cache blocks after evicting is much higher than that of local node cache blocks, prioritizing the retention of cross-node cache blocks reduces the overall computational overhead of the system. At system startup, the system obtains information about the non-consistent memory access node where each graphics processor resides by calling the interface provided by the operating system, and then establishes an affinity mapping table from the graphics processor to the non-consistent memory access node. The mapping table can be stored in memory for subsequent cache allocation and eviction decisions. The establishment of the affinity mapping table is a one-time process, completed during system startup, and does not affect the system's operating performance.

[0030] In this embodiment, the score calculation process for reference-counted cache blocks is as follows: First, a base score is determined based on the normalized value of the most recently accessed timestamp and the affinity coefficient. The base score can be obtained by multiplying the normalized value of the most recently accessed timestamp by the affinity coefficient. Then, the base score is adjusted inversely using the reference count value, so that the higher the reference count value, the lower the score. The inverse adjustment can be done by dividing the base score by the reference count value, or by subtracting a value proportional to the reference count value from the base score. This design results in cache blocks shared by more requests receiving lower scores, making them less likely to be evicted. Evicting a cache block shared by more requests would require more requests to recalculate key-value pairs, increasing system overhead. For example, if a cache block shared by ten requests is evicted, all ten requests would need to recalculate the corresponding key-value pairs, while if a cache block used by only one request is evicted, only one request would need to recalculate. Therefore, cache blocks shared by more requests have higher value and should be retained preferentially.

[0031] In this embodiment, the system sorts all cache blocks included in the eviction candidate according to their scores from highest to lowest. A higher score indicates a lower value for the cache block, and it should be evicted first. The system evicts the cache blocks with the highest scores sequentially until sufficient cache space is freed up. The evicted key-value cache blocks are written to the next-level storage. The next-level storage can be local CPU memory or a non-volatile storage device. When a subsequent request needs to access an evicted cache block, the system reloads it from the next-level storage into the graphics processor's video memory. The system determines the amount of cache space to be freed as follows: When the cache space utilization reaches a preset threshold, the system calculates the amount of space to be freed, which is equal to the currently used space minus the space corresponding to the threshold. Then, the system evicts cache blocks in descending order of scores until the amount of freed space reaches or exceeds the amount of space to be freed. During the eviction process, the system prioritizes evicting the cache blocks with the highest scores, ensuring that only the most valuable cache blocks remain in the cache, thereby maximizing the cache hit rate.

[0032] In an optional embodiment, setting the countdown initial value based on the average reference interval includes: The number of erasure scan cycles experienced by key-value cache blocks in countdown mode within a set time window from entering the countdown to being re-referenced is calculated as the average reference interval, and the initial countdown value is set as the average value.

[0033] In this embodiment, the statistical process for the average reference interval is as follows: The system maintains a time window to record cache block reference events within a recent period. The length of the time window can be adjusted according to the actual needs of the system, typically set to a few minutes to tens of minutes. When a cache block in countdown mode is re-referenced, the system records the number of eligibility scan cycles experienced by the cache block from entering the countdown to being re-referenced. The system stores these cycle counts in a statistical list and periodically calculates the average value. When it is necessary to set the initial countdown value, the system uses this average value as the initial countdown value. If there is insufficient data in the statistical list, a preset default value is used as the initial countdown value. The preset default value can be set according to the historical operating data of the system, typically set to 5 to 10 eligibility scan cycles. The time window is updated using a sliding window method, and each statistical analysis only includes the latest data within the window, thus ensuring that the average reference interval can reflect the current workload characteristics of the system in a timely manner.

[0034] It's important to note that the unit for the average reference interval is the number of eviction scan cycles, rather than time, because the countdown value is updated synchronously with the eviction scan cycle. Each time a cache eviction is triggered, the system decrements the countdown value of all cache blocks in countdown mode. Therefore, the countdown value directly corresponds to the number of eviction scans a cache block can undergo within its buffer period. Using the number of eviction scan cycles as the unit ensures that the initial countdown value remains consistent with the countdown update mechanism, guaranteeing that the buffer period length accurately corresponds to the system-defined average reference interval. If time were used as the unit for the average reference interval, it would require converting time to the corresponding number of eviction scan cycles, increasing system complexity. Furthermore, the length of the eviction scan cycle is dynamic and related to cache space utilization; using the number of eviction scan cycles as the unit better adapts to this dynamic change.

[0035] In this embodiment, the eviction scan cycle refers to the time interval between two cache eviction operations triggered by the system. The system can dynamically adjust the eviction scan cycle based on cache space usage. When cache space utilization is high, the eviction scan cycle is shortened, increasing the frequency of eviction operations. When cache space utilization is low, the eviction scan cycle is extended to reduce unnecessary computational overhead. For example, when cache space utilization exceeds 90%, the system can set the eviction scan cycle to 1 second. When cache space utilization is below 70%, the system can set the eviction scan cycle to 10 seconds. This dynamic adjustment mechanism can reduce system overhead while ensuring sufficient cache space. The dynamic adjustment of the eviction scan cycle is achieved by real-time monitoring of cache space utilization; the system calculates the time for the next eviction scan based on the current utilization.

[0036] For example, suppose the system's time window is 10 minutes long, and in the past 10 minutes, 100 cache blocks in countdown mode were re-referenced. The number of eviction scan cycles these cache blocks experienced from entering countdown to being re-referenced were 3, 5, 7, 8, 2, 4, 6, 9, 1, 5, and so on. The system adds these values ​​together and divides by 100 to obtain an average of 5.2. The system sets the average reference interval to 5 and sets the initial countdown value for newly entered cache blocks to 5. If the subsequent average value changes, for example, if the average reference interval becomes 6 in the next 10 minutes, the system will dynamically update the average reference interval to 6, thus setting the initial countdown value for newly entered cache blocks to 6. This dynamic adjustment method ensures that the initial countdown value always adapts to the current workload characteristics of the system, achieving an optimal balance between reducing false evictions and avoiding unnecessary cache usage.

[0037] In an optional embodiment, when a request matches the key-value cache block, if the reuse metadata field indicates countdown mode, the process switches to reference counting mode and sets the reference count value while updating the most recently accessed timestamp of the key-value cache block.

[0038] In this embodiment, when a cache block in countdown mode is hit again, the system switches it to reference counting mode and sets the reference count value to 1. Simultaneously, the recent access timestamp of the cache block is updated to the current time. This design ensures that the recent access timestamp of the cache block reflects the latest access in subsequent score calculations, resulting in a more accurate score evaluation. If the recent access timestamp is not updated, the cache block may be incorrectly evaluated as a low-value cache block due to its earlier access time, leading to premature eviction. For example, a cache block was accessed 10 seconds ago, then entered countdown mode, and was hit again with 1 second remaining. If the recent access timestamp is not updated, the recent access time of the cache block remains 10 seconds ago, and in the score calculation, it will be considered an earlier accessed cache block, receiving a higher score and being more likely to be evicted. However, if the recent access timestamp is updated to the current time, the cache block will be considered a more recently accessed cache block, receiving a lower score and being less likely to be evicted.

[0039] It's important to note that updating the most recently accessed timestamp is also an atomic operation to ensure data consistency under multi-threaded concurrent access. In scenarios where cache blocks are shared across nodes, when a remote graphics processor hits the cache block, it needs to update the most recently accessed timestamp of that cache block via remote direct memory access to ensure that all graphics processors sharing the cache block see the latest timestamp. If different graphics processors see inconsistent most recently accessed timestamps, it will lead to deviations in score calculation, thus affecting the accuracy of the elimination decision.

[0040] In an optional embodiment, when increasing the reference count in reference counting mode, if the current reference count value reaches a preset upper limit, the reference count value remains unchanged.

[0041] In this embodiment, the upper limit of the reference count value is determined by the number of bits used to store the reference count value in the reused metadata field. Since the remaining seven bits of the reused metadata field are used to store the reference count value, the maximum value of the reference count value is 127. When the reference count value reaches 127, even if a new request hits the cache block, the system will no longer increase the reference count value. This design can prevent the reference count value from overflowing and also avoid score calculation deviations caused by an excessively large reference count value. Because when the reference count value exceeds a certain threshold, continuing to increase the reference count value has little impact on the evaluation of the cache block's value. Setting an upper limit can simplify the system implementation while ensuring the accuracy of the evaluation. For example, a cache block shared by 100 requests and a cache block shared by 200 requests are both highly valuable and should be prioritized for retention. If no upper limit is set for the reference count value, when the reference count value reaches 200, it will cause score calculation overflow or require more bits to store the reference count value, increasing metadata overhead.

[0042] For example, suppose a cache block's reference count has reached 127, and a new request hits that cache block. The system detects that the reference count has reached its limit, so it stops increasing the count and keeps it at 127. The cache block's score calculation still uses 127 as the reference count, resulting in a low score that allows it to be prioritized for retention. Even if more requests hit the cache block subsequently, the reference count will remain at 127, unaffecting its retention priority. This saturation counting method effectively avoids the problem of overflow while ensuring evaluation accuracy.

[0043] In an optional embodiment, the calculation of the score based on the recent access timestamp normalized value, the affinity coefficient of the non-consistent memory access node, and the reference count value includes: The base score is determined based on the normalized value of the most recent access timestamp and the affinity coefficient. The base score is then adjusted in reverse using the reference count value, such that the higher the reference count value, the lower the score.

[0044] In this embodiment, the base score is calculated by multiplying the normalized recent access timestamp value by an affinity coefficient. For example, if a cache block has a normalized recent access timestamp value of 90 and an affinity coefficient of 1.0, the base score is 90. If the affinity coefficient is 0.8, the base score is 72. A reverse adjustment method can be to divide the base score by the reference count value. For example, if the reference count value is 2, the adjusted score is 90 divided by 2, which equals 45. If the reference count value is 10, the adjusted score is 90 divided by 10, which equals 9. Thus, the higher the reference count value, the lower the score, and the less likely the cache block is to be evicted. A reverse adjustment method can also be to subtract a value proportional to the reference count value from the base score. For example, the adjusted score is equal to the base score minus the reference count value multiplied by a preset coefficient. The preset coefficient can be adjusted according to the actual needs of the system, and is usually set between 0.01 and 0.1. Regardless of the reverse adjustment method used, the core idea is to make the reference count value inversely proportional to the score, thereby reflecting the shared value of the cache block.

[0045] In this embodiment, the specific formula for score calculation can be adjusted according to the actual needs of the system. For example, other factors, such as the size of the cache block and the access frequency of the cache block, can be added to the score calculation. However, it should be noted that adding too many factors will increase the complexity of the score calculation and reduce the system's operating efficiency. Therefore, in practical applications, appropriate factors and weights should be selected based on the system's performance requirements and workload characteristics. In this application, the normalized value of the most recent access timestamp, affinity coefficient, and reference count value are selected because they can comprehensively reflect the value of the cache block, while having low computational complexity, making them suitable for application in large-scale key-value caching systems. The calculation of these three factors is very simple, requiring only a small number of arithmetic operations, and will not have a significant impact on the system's operating performance.

[0046] In an optional embodiment, the affinity coefficient is determined as follows: when the non-consistent memory access node where the key-value cache block is located belongs to the same node as the current graphics processor, the affinity coefficient is a first value; when they belong to different nodes, the affinity coefficient is a second value, and the first value is greater than the second value.

[0047] In this embodiment, the selection of the first and second values ​​can be adjusted according to the system's hardware architecture and workload characteristics. For example, the first value can be set to 1.0, and the second value can be set to 0.8. This way, the base score of a cache block on the same node is 1.25 times that of a cross-node cache block. Under the same conditions, cross-node cache blocks have lower scores and are less likely to be evicted. If the latency difference in cross-node memory access is greater in the system, the second value can be appropriately reduced to further improve the retention priority of cross-node cache blocks. For example, if the latency of cross-node memory access is three times that of local node memory access latency, the second value can be set to 0.33. This way, the base score of a cross-node cache block is one-third that of a local node cache block, more accurately reflecting the eviction cost of cross-node cache blocks.

[0048] In this embodiment, the affinity coefficient can also be set in a more granular hierarchical manner. For example, the affinity coefficient can be divided into multiple levels based on the distance between nodes with inconsistent memory access. The closer the nodes are, the higher the affinity coefficient. However, it should be noted that overly granular hierarchies increase system complexity and require more detailed hardware topology information. In practical applications, a two-level affinity coefficient approach usually meets the needs of most scenarios, while being simple to implement and highly efficient. The two-level affinity coefficient approach achieves a good balance between accuracy and complexity, making it suitable for deployment in large-scale heterogeneous server clusters.

[0049] In an optional embodiment, each time a cache eviction is triggered, the countdown value of the key-value cache block that is in countdown mode and whose countdown value has not reached zero is decremented and updated.

[0050] In this embodiment, each time cache eviction is triggered, the system iterates through all cache blocks in countdown mode and decrements their countdown value by 1. If the countdown value reaches zero, the cache block becomes an eviction candidate and participates in subsequent score calculation and eviction sorting. This design synchronizes the decrement of the countdown value with the eviction scan cycle, eliminating the need for a separate timer to maintain the countdown and reducing system overhead. It also ensures that the countdown value accurately reflects the waiting time of the cache block during the eviction scan cycle. Using a separate timer to maintain the countdown would increase the number of system interrupts and context switching overhead, impacting system performance. Synchronizing the decrement of the countdown value with the eviction scan cycle utilizes the existing eviction scan process without requiring additional system resources.

[0051] For example, suppose a cache block enters countdown mode with an initial countdown value of 8. Upon the first cache eviction trigger, the countdown value decreases to 7. Upon the second cache eviction trigger, it decreases to 6. This continues until the eighth cache eviction trigger, when the countdown value reaches 0, and the cache block enters the eviction candidate pool to participate in score calculation and eviction sorting. If, at any of these eight scans, the cache block is hit by a new request, the system immediately switches it back to reference counting mode, stopping the countdown decrement.

[0052] In an optional embodiment, the key-value cache block with the high eviction score further includes: For shared key-value cache blocks across nodes with inconsistent memory access, the remote graphics processor sharing the key-value cache block is notified via remote direct memory access to confirm that there are no references before the eviction process is performed.

[0053] In this embodiment, in a multi-GPU system, a key-value cache block may be shared by multiple GPUs. When a GPU needs to evict a cross-node shared key-value cache block, it must ensure that other GPUs sharing the cache block are no longer using it. Otherwise, other GPUs accessing the cache block will find it has been evicted and need to recalculate, increasing system overhead. Therefore, before evicting a cross-node shared key-value cache block, the system notifies other GPUs sharing the cache block via remote direct memory access to confirm whether these GPUs are still referencing the cache block. Only after all GPUs sharing the cache block confirm that there are no references will the system perform the eviction operation.

[0054] In this embodiment, Remote Direct Memory Access (RDA) technology allows different nodes to directly access each other's memory without requiring data copying through the central processing unit, resulting in higher transmission efficiency. Sending notifications and receiving acknowledgments via RDA reduces the overhead of cross-node communication and improves system response speed. Traditional network communication methods increase the overhead of data copying and protocol processing, leading to increased latency in cross-node communication. RDA, however, bypasses the operating system kernel and performs data transmission directly at the hardware level, resulting in lower latency and higher bandwidth. This technology is well-suited for cross-node small message notification scenarios, ensuring the correctness of cache block eviction operations without impacting system performance.

[0055] In this embodiment, the system maintains a shared cache block list, recording which graphics processors (GPUs) share each cache block. Each cache block has a unique identifier used to distinguish different cache blocks in the system. When a GPU needs to evict a cache block, it first checks if the cache block is in the shared cache block list. If it is, it notifies other GPUs in the list to confirm that there are no references. If it is not, the eviction operation can be performed directly. The shared cache block list can be stored in the local memory of each GPU or in a global shared memory area. In practice, the shared cache block list is usually stored in the local memory of each GPU to improve access speed. When a cache block is shared by multiple GPUs, each GPU's local shared cache block list records the information of the cache block and the GPU IDs of other GPUs sharing the cache block.

[0056] In an optional embodiment, the notification to the graphics processor sharing the key-value cache block confirming no reference includes: A query message is sent to the remote graphics processor that shares the key-value cache block. Only after receiving a no-reference confirmation from the remote graphics processor is the key-value cache block marked as ready for removal.

[0057] In this embodiment, when a key-value cache block shared across nodes needs to be evicted, the local graphics processor sends a query message to all remote graphics processors sharing the cache block. The query message contains the identifier information of the cache block. After receiving the query message, each remote graphics processor checks whether it still references the cache block. If it does not reference it, it returns a no-reference confirmation message to the local graphics processor. After receiving all no-reference confirmation messages from the remote graphics processors, the local graphics processor marks the cache block as eligible for eviction and performs the eviction operation.

[0058] In this embodiment, the system sets a timeout period. If no acknowledgment message is received from a remote graphics processor within the timeout period, the local graphics processor can resend the query message or temporarily retain the cached block, waiting for subsequent replacement scan cycles to process it. This design can avoid erroneous replacements due to network latency or remote graphics processor failure. The length of the timeout period can be adjusted according to the system's network latency and reliability, typically set between a few milliseconds and tens of milliseconds. If the timeout period is set too long, it will increase the delay of the replacement operation, affecting system performance. If the timeout period is set too short, it will lead to frequent retransmissions, increasing network overhead. In practical applications, the system dynamically adjusts the timeout period based on historical network latency data to ensure that the delay of the replacement operation is minimized while maintaining reliability.

[0059] Based on the above embodiments of a non-consistent memory access-aware key-value cache hierarchical scheduling method, this application also provides a non-consistent memory access-aware key-value cache hierarchical scheduling system, such as... Figure 2 As shown, it includes at least one processor and a memory, the memory storing instructions that, when executed by the processor, cause the system to perform the non-consistent memory access-aware key-value cache hierarchical scheduling method as described in the above embodiments.

[0060] In this embodiment, the processor can be a central processing unit (CPU) or a graphics processing unit (GPU). The memory stores instructions and data, including key-value cache blocks and corresponding metadata. When an instruction is executed by the processor, it causes the system to perform the method described in any of the preceding claims. Deploying the system on heterogeneous server nodes with multiple CPUs and GPUs fully utilizes hardware resources and improves the performance of large model inference. The system may also include a network interface for communicating with other server nodes to achieve distributed large model inference.

[0061] It should be noted that the system in this application is fully compatible with the existing non-consistent memory access-aware key-value cache hierarchical scheduling framework. No modifications are required to the underlying paging mechanism, memory allocation mechanism, hardware scheduling logic, or input / output paths. Only the metadata structure and eviction decision logic of the key-value cache blocks need to be modified, resulting in extremely low implementation costs. The system can be directly integrated into existing large-scale model inference engines without requiring large-scale modifications to existing systems. This compatibility design enables the technical solution of this application to be quickly deployed and applied, bringing performance improvements to existing systems.

[0062] For example, the technical solution of this application will be described in detail below with reference to a specific system operation process. The system is deployed on a server containing two non-uniform memory access nodes and four graphics processors. Each node contains two graphics processors and corresponding local CPU memory. When the system starts, it first probes the non-uniform memory access topology of the server and establishes an affinity mapping table from graphics processors to non-uniform memory access nodes. The probe process is implemented by calling the interface provided by the operating system, which can obtain the physical slot where each graphics processor is located and the corresponding non-uniform memory access node number. Then, a key-value cache space of a certain size is allocated to each graphics processor. The size of the cache space can be adjusted according to the graphics processor's video memory capacity, usually set to 60% to 80% of the video memory capacity. At the same time, the metadata of each cache block is initialized, the reuse metadata field is set to 0, the mode bit is set to 0, and the countdown value is 0, which has no practical meaning. The most recently accessed timestamp is set to the system startup time.

[0063] During system operation, multiple inference requests are received. Each inference request contains input text and generation parameters. The system first preprocesses the input text, converting it into a token sequence. Then, it enters a pre-filling stage, calculating key-value pairs for the input text and storing them in a key-value cache. After the pre-filling stage is complete, it enters a decoding stage, generating output text token by token. During the decoding stage, the generation of each token requires accessing the previously generated key-value cache block.

[0064] When the first user request enters the system, the system calculates the prefix hash value of the input text, but no matching cache block is found in the global index table. Therefore, a pre-filling calculation is triggered, generating 15 system prefix cache blocks and 1 user-private cache block. The reuse metadata field of all cache blocks is set to 129, which means the mode bit is 1 and the reference count is 1. The last access timestamp is set to the current time, and the normalized value is 100.

[0065] Ten subsequent user requests entered the system, all containing the same system prefix in their input text. After calculating the prefix hash, the system matched it against the 15 previously generated system prefix cache blocks. For each matched cache block, the system checked its reuse metadata field and found it to be in reference counting mode, thus incrementing the reference count by 1. After matching 10 requests, the reference count for each system prefix cache block became 11, and the value of the reuse metadata field was 139.

[0066] When cache space utilization reaches 90%, the system triggers a cache eviction operation. The system first iterates through all cache blocks in countdown mode, decrementing their countdown value by 1. At this point, there are no cache blocks in countdown mode, so this step is skipped. Next, the system calculates a score for all cache blocks. For system prefix cache blocks, the recent access timestamp normalization value is 90, the affinity coefficient is 1.0, and the reference count is 11. The base score is 90 multiplied by 1.0 equals 90. The adjusted score is 90 divided by 11, approximately equal to 8.18. For user-private cache blocks, the recent access timestamp normalization value is 90, the affinity coefficient is 1.0, and the reference count is 1. The score is 90 multiplied by 1.0 divided by 1 equals 90. The system sorts all cache blocks from highest to lowest score, with user-private cache blocks having the highest score and ranking first. The system evicts the highest-scoring user-private cold blocks sequentially until enough cache space is freed up. System prefix cache blocks have very low scores and are not evicted.

[0067] After all 11 requests have been processed, each request generates an end-of-request token, ending the inference process. The system decrements the reference count of each system prefix cache block by 1 sequentially. When the last request completes, the reference count reaches 0. The system then switches the mode bit of these cache blocks to 0, entering countdown mode. At this point, the system's average reference interval is 8 eviction scan cycles, so the initial countdown value is set to 8. The value of the reuse metadata field becomes 8.

[0068] Three subsequent cache eviction scans were triggered. During each scan, the system decremented the countdown value of the system prefix cache blocks in countdown mode by 1. After three scans, the countdown value became 5, which was not yet 0, so these cache blocks were not included in the eviction candidate pool.

[0069] Before the fourth scan, a new user request enters the system, its input text containing the same system prefix. After calculating the prefix hash value, the system matches a system prefix cache block in countdown mode. The system switches the mode bit to 1, entering reference counting mode, and sets the reference count value to 1. Simultaneously, the last access timestamp is updated to the current time, and the normalized value is 100. The value of the reuse metadata field becomes 129. These cache blocks do not need to be re-pre-filled and are directly reused by the new request.

[0070] If a graphics processor located on the second non-consistent memory access node initiates a request, it matches the system prefix cache block on the first node. The system increments the reference count of that cache block by 1, making it 2. The affinity coefficient is 0.8 when calculating the score. The base score is 90 multiplied by 0.8 equals 72. The adjusted score is 72 divided by 2, equaling 36. This score is significantly lower than the score of 90 for a local private cache block, therefore this cross-node shared cache block will be preferentially retained and will not be easily evicted.

[0071] If no new requests reference the system's prefix cache block in the subsequent eight consecutive scans, the countdown value will decrease sequentially to 7, 6, 5, 4, 3, 2, 1, and 0. When the countdown value reaches 0, the cache block is marked as eligible for eviction and enters the eviction candidate pool. The system sorts it together with other private cold blocks according to their scores, with higher-scoring blocks being evicted first. Evicted cache blocks are written to the next-level non-volatile storage device via the batch input / output interface.

[0072] In this embodiment, existing cache eviction policies are primarily based on Least Recently Used (LRU) sorting. All cache blocks, regardless of their sharing level or location on the storage node, are evicted according to a uniform access time order. This strategy performs well when handling private cache blocks, but has significant limitations when handling shared and cross-node cache blocks. For shared cache blocks, the LRU strategy cannot distinguish between high-value and low-value shared blocks, potentially leading to the wrongful eviction of high-value blocks. For example, a cache block shared by multiple requests, if not accessed for a period, might be considered a low-value cache block by the LRU strategy and thus prioritized for eviction. However, this cache block may be reused by multiple requests later, and eviction would result in significant recalculation overhead. For cross-node cache blocks, the LRU strategy does not consider the high latency cost of cross-node memory access, potentially leading to frequent eviction and reloading of cross-node cache blocks, increasing system overhead. For example, if a cross-node cache block and a local node cache block have the same recent access time, the LRU strategy might treat them as cache blocks of equal value, potentially prioritizing the eviction of the cross-node cache block. However, in reality, the overhead of recalculating after a cross-node cache block is evicted is much higher than that of a local node cache block. Prioritizing the evicting of cross-node cache blocks will lead to a decrease in the overall performance of the system.

[0073] Some existing technologies set a fixed percentage of reserved space for shared cache blocks to protect them from eviction within a certain area. However, the dependency levels of different shared cache blocks vary greatly, and a fixed reserved space cannot be dynamically adjusted according to the actual value of the cache blocks. A fixed reserved space may allow low-value shared blocks to occupy space for a long time, wasting cache resources, or it may mistakenly evict high-value shared blocks outside the reserved space, affecting system performance. For example, if the reserved space ratio is set to 20%, when there are a large number of low-value shared blocks in the system, these low-value shared blocks will occupy the entire reserved space, while high-value shared blocks can only reside outside the reserved space and are easily evicted. When the number of shared cache blocks in the system is small, the space in the reserved space will be wasted and cannot be used to store private cache blocks, reducing the utilization rate of cache space.

[0074] Other existing technologies use reference counting to manage shared cache blocks, immediately evicting them when the reference count reaches zero. However, references to shared cache blocks are often intermittent; their reference counts may briefly drop to zero between requests, but this doesn't mean there will be no subsequent references. Directly evicting cache blocks with zero reference counts leads to a large amount of unnecessary recalculation, increasing decoding latency and system load. For example, in a multi-turn conversation scenario, a user might resume a previous conversation after a period of time, requiring access to previously generated key-value cache blocks. If these cache blocks' reference counts drop to zero and they are evicted during conversation breaks, the system needs to recalculate all key-value pairs when the user resumes the conversation, resulting in increased response latency.

[0075] Furthermore, existing technologies typically use separate fields to store reference counts and countdown information, leading to increased metadata overhead. In large-scale key-value caching systems, the number of cache blocks can reach millions or even tens of millions. Increased metadata overhead consumes significant memory resources, impacting system scalability. For example, if each cache block uses two bytes to store reference counts and countdown information separately, for ten million cache blocks, the metadata overhead would reach twenty megabytes. This application, however, reduces metadata overhead by half by reusing a single-byte metadata field, resulting in substantial memory savings for large-scale key-value caching systems.

[0076] This application achieves switching between reference counting and countdown modes without increasing metadata overhead by reusing a one-byte metadata field. Reference counting mode records the current dependency level of cache blocks, accurately reflecting their shared value. Countdown mode provides a buffer period for cache blocks whose reference count reaches zero, preventing false evictions due to intermittent references. The combination of these two modes simultaneously solves both the value assessment problem of shared cache blocks and the problem of false evictions caused by intermittent references.

[0077] Furthermore, this application combines the recent access timestamp, affinity coefficient, and reference count to form a comprehensive scoring and evaluation mechanism. This mechanism can simultaneously consider the access frequency, sharing degree, and cross-node eviction cost of cache blocks, thus more accurately assessing the actual value of cache blocks. The recent access timestamp reflects the recent access status of the cache block, reflecting its temporal locality. The affinity coefficient reflects the affinity between the node where the cache block resides and the current graphics processor, reflecting the eviction cost of cross-node cache blocks. The reference count reflects the degree of dependency on the cache block, reflecting its shared value. The organic combination of these three factors enables the scoring and evaluation mechanism to comprehensively and accurately reflect the value of cache blocks, thereby making more reasonable eviction decisions.

[0078] In the score calculation, this application reverses the reference count, resulting in lower scores for cache blocks shared by more requests, making them less likely to be evicted. This design prioritizes the retention of high-value shared cache blocks, reducing the recalculation overhead caused by the evicting of shared cache blocks. Simultaneously, this application differentiates the eviction costs of local and cross-node cache blocks using an affinity coefficient, ensuring that cross-node cache blocks receive lower scores and are prioritized for retention. This design effectively reduces the overhead of cross-node recalculation, improving the overall system performance.

[0079] Furthermore, this application sets the countdown initial value by dynamically calculating the average reference interval, which can adapt to different workload characteristics. The average reference interval reflects the average reference gap between cache blocks in the system. The countdown initial value set based on the average reference interval can strike a balance between reducing false evictions and avoiding unnecessary cache occupation. When the workload concurrency is high, the reference gap between cache blocks is short, the average reference interval is small, and the countdown initial value is also small, which can release cache blocks that are no longer in use in a timely manner. When the workload concurrency is low, the reference gap between cache blocks is long, the average reference interval is large, and the countdown initial value is also large, which can reduce false evictions caused by intermittent references.

[0080] This application also optimizes the eviction process for cross-node shared cache blocks by notifying the remote graphics processor via remote direct memory access to confirm that no references are being used before performing the eviction operation. This design avoids system errors and performance degradation caused by mistakenly evicting cache blocks that are being used by other graphics processors. Furthermore, using remote direct memory access technology for cross-node communication reduces communication overhead and improves system response speed.

[0081] All the technical features in this application form a complete logical loop. First, through a metadata design that reuses byte bit fields, the contradiction between dynamic scheduling and low metadata overhead is resolved, providing a feasible storage foundation for upper-layer logic. Then, through a weighted scoring mechanism that uses reference counting and inconsistent memory access, the problems of static priority and hard-isolated reserved areas are solved, accurately quantifying the shared value of cache blocks and the cost of cross-node eviction, enabling dynamic space contention between shared and private cache blocks. Finally, through a countdown buffer mechanism, the problem of false eviction caused by intermittent references is solved, further improving the cache hit rate. These three layers of technical features work together to effectively improve the throughput of large-model inference systems and reduce inference latency.

[0082] While existing technologies include schemes that use reference counting to manage shared caches, schemes that use countdown buffers to avoid false evictions, and memory management schemes that consider inconsistent memory access architectures, none combine these three features to form a unified, low-overhead key-value cache eviction strategy suitable for large-scale model inference with inconsistent memory access architectures. Reference counting schemes, while quantifying shared value, do not consider the differences in inconsistent memory access architectures or address the problem of false eviction due to intermittent references. Countdown buffer schemes, while avoiding false evictions, also do not quantify shared value or consider the differences in inconsistent memory access architectures. Inconsistent memory access-aware memory management schemes, while considering the differences in cross-node access, are not optimized for shared key-value caches in large-scale model inference and do not address the problem of intermittent references. This application organically combines these features to form a novel technical solution that can simultaneously solve multiple problems existing in existing technologies.

[0083] Those skilled in the art will understand that the technical solution of this application can be applied to various large-scale model inference scenarios, including but not limited to text generation, dialogue systems, machine translation, and image generation. The technical solution of this application can also be applied to other scenarios requiring the management of shared caches, such as database systems and distributed storage systems. This application is not limited to these, and will not be elaborated upon here.

[0084] It should be noted that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A hierarchical scheduling method for non-consistent memory access-aware key-value caches, characterized in that, include: When a request matches a key-value cache block, if the reuse metadata field indicates reference counting mode, the reference count of the key-value cache block is increased; if it indicates countdown mode, the reference count is switched to reference counting mode and the reference count value is set. When the request ends and the key-value cache block is no longer used, the reference count is decreased, and when the reference count decreases to zero, the countdown mode is switched to countdown mode and an initial countdown value is set based on the average reference interval. For key-value cache blocks in reference counting mode, a score is calculated based on the normalized value of the most recent access timestamp, the affinity coefficient of the non-consistent memory access node, and the reference count value; key-value cache blocks whose countdown values ​​have not reached zero are not included in the eviction candidate. For key-value cache blocks whose countdown values ​​have reached zero, a score is calculated based on their most recently accessed timestamp normalized value and affinity coefficient. The key-value cache block with the highest score is evicted and written to the next level of storage; the reused metadata field is one byte, of which one bit is the mode bit and the remaining bits store the reference count value or countdown value; the affinity coefficient is determined based on the affinity relationship between the non-consistent memory access node where the key-value cache block is located and the current graphics processor.

2. The method according to claim 1, characterized in that, The method of setting the countdown initial value based on the average reference interval includes: The number of erasure scan cycles experienced by key-value cache blocks in countdown mode within a set time window from entering the countdown to being re-referenced is calculated as the average reference interval, and the initial countdown value is set as the average value.

3. The method according to claim 1, characterized in that, When a request matches the key-value cache block, if the reuse metadata field indicates countdown mode, the request is switched to reference counting mode and the reference count value is set. At the same time, the recent access timestamp of the key-value cache block is updated.

4. The method according to claim 1, characterized in that, When increasing the reference count in reference counting mode, if the current reference count value reaches the preset upper limit, the reference count value remains unchanged.

5. The method according to claim 1, characterized in that, The score is calculated based on the normalized value of the most recent access timestamp, the affinity coefficient of the non-consistent memory access node, and the reference count value, including: The base score is determined based on the normalized value of the most recent access timestamp and the affinity coefficient. The base score is then adjusted in reverse using the reference count value, such that the higher the reference count value, the lower the score.

6. The method according to claim 1, characterized in that, The affinity coefficient is determined as follows: when the non-consistent memory access node where the key-value cache block is located belongs to the same node as the current graphics processor, the affinity coefficient is a first value; when they belong to different nodes, the affinity coefficient is a second value, and the first value is greater than the second value.

7. The method according to claim 1, characterized in that, Also includes: Each time a cache eviction is triggered, the countdown value of the key-value cache block that is in countdown mode and whose countdown value has not reached zero is decremented and updated.

8. The method according to claim 1, characterized in that, The key-value cache block with the highest elimination score also includes: For shared key-value cache blocks across nodes with inconsistent memory access, the remote graphics processor sharing the key-value cache block is notified via remote direct memory access to confirm that there are no references before the eviction process is performed.

9. The method according to claim 8, characterized in that, The notification that the graphics processor sharing the key-value cache block confirms no reference includes: A query message is sent to the remote graphics processor that shares the key-value cache block. Only after receiving a no-reference confirmation from the remote graphics processor is the key-value cache block marked as ready for removal.

10. A non-consistent memory access-aware key-value cache hierarchical scheduling system, characterized in that, The system includes at least one processor and a memory, the memory storing instructions that, when executed by the processor, cause the system to perform the method as described in any one of claims 1 to 9.

Images