A cache data scheduling method and an electronic device

Abstract

The application discloses a cache data scheduling method and an electronic device, and relates to the technical field of computers. The method comprises the following steps: when the occupancy rate of accelerator video memory or processor memory reaches a first preset threshold, determining the priority score of a current storage node of the accelerator video memory or the processor memory according to the access frequency, recency and depth in the prefix cache tree of the current storage node; selecting a target node to be evicted from the current storage node according to the priority score, removing the physical address mapping of the target node in the accelerator video memory or the processor memory, marking the target node as a ghost node, and reserving the metadata of the ghost node in the processor memory; periodically scanning the ghost node, selecting a target ghost node to be prefetched from the ghost node according to the priority score of the ghost node, reading the data cache block corresponding to the target ghost node from the computing fast link extended memory, and loading the data cache block into the processor memory. The application improves the cache hit rate.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a cached data scheduling method and an electronic device. Background Technology

[0002] In long-text inference scenarios using pre-trained language models, the GPU memory consumption of data caching is enormous. Typically, a hierarchical storage architecture, including accelerator memory, processor memory, and Compute Express Link (CXL) extended memory, is needed to expand capacity. In related technologies, the Least Recently Used (LRU) strategy primarily determines cache eviction based on access frequency or time recentity. This can lead to the system mistakenly evicting critical data blocks that have low recent access frequency. Once these critical data blocks are removed, their subordinate data blocks become unusable due to broken dependency chains, forcing the system to perform costly recomputation in subsequent inferences, severely reducing inference efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a cache data scheduling method and an electronic device, which effectively improves the cache hit rate.

[0004] To achieve the above objectives, this invention provides a data caching method applied to a storage architecture comprising accelerator memory, processor memory, and compute fast link extended memory. The method includes: when the occupancy rate of accelerator memory or processor memory reaches a first preset threshold, determining the priority score of the current storage node based on its access frequency, proximity, and depth in the prefix cache tree; wherein the prefix cache tree is a tree-like data structure used to organize contextual dependencies between data cache blocks, and proximity characterizes the time proximity of access to the current storage node; selecting a target node to be evicted from the current storage nodes based on the priority score, removing the physical address mapping of the target node in the accelerator memory or processor memory, marking the target node as a ghost node, and retaining the ghost node's metadata in the processor memory; periodically scanning ghost nodes, determining their priority score based on their access frequency, proximity, and depth in the prefix cache tree, selecting a target ghost node to be prefetched from the ghost nodes based on the priority score, reading the data cache block corresponding to the target ghost node from the compute fast link extended memory, and loading it into the processor memory.

[0005] To achieve the above objectives, the present invention provides a data caching device applied in a storage architecture including accelerator memory, processor memory, and compute fast link extended memory. The device includes: a determining module, configured to determine a priority score of the current storage node based on its access frequency, proximity, and depth in the prefix cache tree when the occupancy rate of the accelerator memory or processor memory reaches a first preset threshold; wherein the prefix cache tree is a tree-like data structure for organizing contextual dependencies between data cache blocks, and proximity characterizes the proximity of the access times of the current storage node; eviction. The first module is used to select target nodes to be evicted from the current storage nodes based on priority scores, remove the physical address mapping of the target nodes in the accelerator memory or processor memory, mark the target nodes as ghost nodes, and retain the metadata of the ghost nodes in the processor memory. The second module is used to periodically scan ghost nodes, determine the priority score of ghost nodes based on the access frequency, proximity, and depth in the prefix cache tree, select target ghost nodes to be prefetched from the ghost nodes based on the priority scores, read the data cache block corresponding to the target ghost node from the compute fast link extended memory, and load it into the processor memory.

[0006] To achieve the above objectives, the present invention provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the cached data method described above when executing the computer program.

[0007] To achieve the above objectives, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cached data method described above.

[0008] To achieve the above objectives, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the cached data method described above.

[0009] The cached data scheduling method provided by this invention introduces a multi-dimensional priority scoring mechanism based on prefix cache tree depth, access frequency, and proximity. Specifically, when the accelerator memory or processor memory occupancy reaches a threshold, not only is the data's accessibility considered, but also the data's depth in the prefix cache tree is incorporated into the scoring system. Since the prefix cache tree reflects the contextual dependencies between data blocks, shallower nodes typically represent more fundamental shared prefixes. By comprehensively considering these three indicators to determine priority scores and select target nodes to be evicted, this method can identify and retain critical nodes that, although not frequently accessed in the short term, are crucial for maintaining the integrity of the dependency chain, thus avoiding dependency chain breaks caused by the accidental deletion of critical nodes. Therefore, this invention significantly reduces the probability of recomputation caused by cache misses, improves cache hit rate, and thereby enhances the overall computational efficiency and resource utilization of large model inference. Furthermore, when evicting a target node, this invention does not simply discard all information, but rather removes its physical address mapping, marks the target node as a ghost node, and retains the ghost node's metadata in the processor memory, enabling the system to maintain global awareness of evicted data with extremely low memory overhead. By periodically scanning these ghost nodes and recalculating priority scores using access frequency, recentity, and tree depth, high-value ghost nodes most likely to be accessed again can be accurately identified from massive amounts of evicted data. Subsequently, this data is proactively read from the extended memory of the fast computing link and loaded into the processor memory. This proactive prefetching mechanism based on ghost node metadata transforms the originally passive, blocking on-demand loading into an active, non-blocking data preparation process. Therefore, this invention effectively masks the high latency of reading data from extended memory, reduces the time inference requests wait for data loading, significantly reduces first-word latency, and ensures the response speed and stability of long text inference services. This invention also discloses a data caching device, an electronic device, a computer-readable storage medium, and a computer program product, which can achieve the same technical effects.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0011] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a data caching method according to an exemplary embodiment.

[0013] Figure 2 This is a flowchart illustrating another method for caching data according to an exemplary embodiment.

[0014] Figure 3 This is a system architecture diagram for one application embodiment of the present invention.

[0015] Figure 4 This is a structural diagram of a prefix cache tree according to an exemplary embodiment.

[0016] Figure 5 This is a schematic diagram illustrating the results of comparison one in one application embodiment of the present invention.

[0017] Figure 6 This is a schematic diagram illustrating the results of comparison two in one application embodiment of the present invention.

[0018] Figure 7 This is a structural diagram of a data caching device according to an exemplary embodiment.

[0019] Figure 8 This is a structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0021] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] Pre-trained language models generally refer to language model training tasks designed based on large-scale corpora (including language training materials such as sentences and paragraphs). A large-scale neural network algorithm structure is trained to learn and implement the model, resulting in the pre-trained language model. Subsequent tasks can then use this model for feature extraction or task fine-tuning to achieve specific objectives. The idea behind pre-training is to first train a set of model parameters for one task, then use these parameters to initialize the network model parameters, and finally use the initialized network model to train other tasks, obtaining models adapted for those tasks. By pre-training on large-scale corpora, neural language representation models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text. Pre-trained language models can provide tokens containing rich semantic information and sentence-level features for downstream tasks. Fine-tuning can also be performed directly on the pre-trained model for downstream tasks, conveniently and quickly obtaining downstream-specific models.

[0023] The neural network algorithm structure used to train a pre-trained language model can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc., or it can be a model built with attention networks, such as Transformer (Self-Attention Neural Network), BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), Clip (Contrastive Language–Image Pre-training), etc. This invention does not limit the specific architecture. An attention network is a network model trained using an attention mechanism. This model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, resulting in a more accurate output.

[0024] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] This invention provides a data caching method, and the method is described in detail below in conjunction with its execution flow. See also... Figure 1 The flowchart illustrates a data caching method according to an exemplary embodiment.

[0026] S101: When the occupancy rate of accelerator memory or processor memory reaches the first preset threshold, the priority score of the current storage node is determined based on the access frequency, proximity, and depth in the prefix cache tree of the current storage node of accelerator memory or processor memory; wherein, the prefix cache tree is a tree data structure used to organize the contextual dependencies between data cache blocks, and proximity is used to characterize the proximity of the time when the current storage node is accessed.

[0027] Accelerator memory refers to the high-speed storage area with high bandwidth integrated within dedicated computing chips such as GPUs (Graphics Processing Units). It is mainly used to store model weights and key-value caches currently participating in computation. Processor memory refers to the system's main memory that can be directly addressed by the Central Processing Unit (CPU). It typically has a larger capacity than accelerator memory but lower bandwidth. Compute Fast Link Extended Memory (CXL) refers to a scalable memory pool connected via the CXL protocol. It has a large capacity and relatively low cost, but its access latency is higher than the former two. A prefix cache tree is a tree-like index structure where nodes represent different text prefix sequences, parent-child node relationships represent text continuation relationships, and the tree depth represents the length or level of the prefix. The root node has the smallest depth, and the leaf nodes have the largest depth. Nodes in accelerator memory are Tier-1 (active nodes), nodes in processor memory are Tier-2 (active nodes), and nodes in compute fast link extended memory are Tier-3 (ghost nodes).

[0028] In this step, the usage of accelerator memory and processor memory is monitored in real time. Once the used capacity of any storage medium reaches a first preset threshold, memory resources are considered strained, and cache scheduling needs to be performed. At this point, all storage nodes currently residing on these two media are traversed, and three key characteristics of each node are extracted: historical access frequency, last access time (used to calculate recentity), and its level depth in the prefix cache tree. A priority score is then generated, which is used to comprehensively evaluate the importance of the current storage node. The lower the score, the more suitable the node is to be evicted.

[0029] As can be seen, this step, by introducing the structural feature of prefix cache tree depth, breaks through the limitations of traditional methods that rely solely on time and frequency. It can identify data that is in a critical position in the tree structure, avoid the failure of a large number of subsequent caches that depend on the prefix due to the accidental deletion of the critical prefix, and thus make a more intelligent retention decision when memory is tight.

[0030] As a feasible implementation method, priority scoring of node n The calculation formula is: ;in, Let n be the frequency of node n's visits. Let n be the proximity of node n. Let n be the depth of node n in the prefix cache tree. , These are the weighting coefficients. The exponential coefficient is used when selecting target nodes to be expelled. It is a negative value.

[0031] For data residing in high-speed memory (levels 1 and 2), the goal is to preserve the skeleton of the radix tree. (Settings...) Convert the structure item to This penalizes long nodes (those consuming more memory per unit of utility, ensuring a rapid decrease in GPU / memory usage) and tends to preserve shorter shared prefixes closer to the root node (such as system hints), thereby maximizing the hit rate of different requests. In actual deployments, when At that time, you can set .

[0032] S102: Select the target node to be evicted from the current storage nodes according to the priority score, remove the physical address mapping of the target node in the accelerator video memory or processor memory, mark the target node as a ghost node, and retain the metadata of the ghost node in the processor memory.

[0033] In this step, nodes that meet the eviction criteria (e.g., lowest score) are selected from all current storage nodes based on priority scores as target nodes. Next, the memory management interface is invoked to release the physical address space occupied by these target nodes in the accelerator's video memory or processor memory, and their status is marked as free to allow new data to be stored. Crucially, the information of these nodes is not completely discarded; instead, their core metadata is extracted, and the metadata of the ghost nodes is retained in the processor memory. This can be understood as an index card, recording the identity and popularity information of the original data block, but no longer holding the large data entity.

[0034] As can be seen, this step, by preserving the metadata of ghost nodes, achieves global awareness of evicted data with minimal memory overhead. This not only avoids the cold start problem caused by the complete loss of historical access information but also provides a data foundation for subsequent intelligent prefetching, effectively balancing the contradiction between the capacity limitations of high-speed memory and cache hit rate.

[0035] As a feasible implementation method, selecting target nodes to be evicted from the current storage nodes based on priority scores includes: sorting the current storage nodes from smallest to largest according to priority scores, and selecting the top M current storage nodes as target nodes to be evicted from the sorting results.

[0036] In practice, the priority scores of all candidate nodes are put into a list, and an ascending sorting algorithm is executed. After sorting, the node at the head of the list is the object that should be removed. The first M elements of the list are extracted and marked as the set of target nodes to be removed.

[0037] As a preferred implementation, the process of sorting the current storage nodes from smallest to largest according to their priority scores further includes: when there are two or more current storage nodes with the same priority score, sorting them from largest to smallest according to their depth in the prefix cache tree.

[0038] In practice, a second-level judgment logic is added to the comparison function of the sorting algorithm. When comparing two nodes A and B, their priority scores are compared first. If the scores are different, they are sorted according to their scores; if the scores are exactly the same, their depths in the prefix cache tree are compared. Nodes with greater depths are considered more likely to be evicted and are thus selected for eviction at a higher position.

[0039] As can be seen, introducing depth as a secondary sorting criterion further refines the granularity of the eviction strategy. In cases of fuzzy scoring, leveraging the topological characteristics of the tree structure to make decisions more consistent with the reasoning patterns of large models maximizes the protection of highly shared shallow prefixes and improves the overall robustness of the caching system.

[0040] As a feasible implementation, the target node is marked as a ghost node, and the metadata of the ghost node is retained in the processor memory, including: obtaining the complete prefix hash value of the target node; mapping the complete prefix hash value to a short fingerprint; wherein the number of bits in the short fingerprint is less than the number of bits in the complete prefix hash value; encoding the access frequency, recentity, and depth in the prefix cache tree of the target node into a compressed load; associating the short fingerprint with the compressed load and storing it in a shadow cache based on a cuckoo hash structure; and marking the target node as a ghost node.

[0041] The complete prefix hash value is a long binary string generated by hashing the original text prefix content, used to uniquely identify the data block. A short fingerprint is a shorter binary string (e.g., 8 bits) generated by truncating or rehashing the complete hash value, aiming to reduce storage space usage. Compressed load refers to compressing multiple scalar features such as access frequency, recentity, and depth into a compact data unit using bit packing or encoding techniques. Cuckoo hash is an efficient hash table implementation that allows storing multiple candidate items in each slot and resolving collisions through a kick-out mechanism, achieving constant-time lookup performance and extremely high space utilization. Shadow cache refers to a memory area specifically used to store ghost node metadata.

[0042] In practice, the complete prefix hash value of the target node is first calculated. To achieve lightweight storage of ghost nodes, long hashes are not stored directly; instead, they are mapped to short fingerprints. Next, the node's statistical characteristics (frequency, recentity, depth) are encoded into a compressed payload in the form of an integer or byte array. Finally, the short fingerprint is used as the key, and the compressed payload as the value, and inserted into a shadow cache table built based on the Cuckoo Hash algorithm. The high space utilization of Cuckoo Hash is ideal for storing massive numbers of ghost nodes because it can accommodate more entries with limited memory and offers extremely fast lookup speeds.

[0043] As can be seen, the storage method combining short fingerprints, compressed load balancing, and cuckoo hashing retains the statistical characteristics of a ghost node in the shadow cache even when it is completely removed from the tree structure under memory pressure. When the system scans for ghost nodes or processes new requests, it queries this structure. Once a fingerprint matches, historical hit counts and recency can be retrieved immediately, ensuring that recurring prefixes can quickly recover their priority using historical statistics. This approach offers higher stability compared to strategies based solely on timeliness.

[0044] S103: Periodically scan ghost nodes, determine the priority score of ghost nodes based on their access frequency, proximity, and depth in the prefix cache tree, select target ghost nodes to be prefetched from the ghost nodes based on the priority score, read the data cache block corresponding to the target ghost node from the compute fast link extended memory, and load it into the processor memory.

[0045] In this step, a background thread is started to scan the list of ghost nodes in the shadow cache at preset time intervals or preset request processing numbers. For each ghost node, its recorded access frequency, recentity, and depth information are read, and a priority score is calculated using the priority scoring algorithm described above. The ghost node with the highest priority score is selected as the target ghost node to be prefetched. It should be noted that when selecting the target ghost node to be prefetched... It is a positive value. Active prefetching ( For data residing in a high-latency layer (Level 3), the goal is to amortize I / O (Input / Output) overhead. (Settings) This transforms the structure term into a linear reward. This tends to prefetch longer sequences because the seek latency for retrieving long nodes is similar to that for short nodes, but it recovers more tokens, achieving a higher ROI on I / O. In practical deployments, when... At time, fixed And adopt dynamic Defined as Subsequently, an asynchronous I / O request is initiated to read the actual data blocks corresponding to these target ghost nodes from CXL memory and load them into the free area of ​​the processor memory. At the same time, the mapping relationship is updated to prepare to respond to possible access requests at any time.

[0046] As can be seen, this step leverages the temporal and spatial locality of data access by transforming passive waiting for access into proactive predictive loading. By periodically evaluating the value of ghost nodes and relocating high-value data in advance, the high latency of the underlying extended memory is effectively masked, significantly reducing pause times during inference.

[0047] As a feasible implementation method, selecting target ghost nodes to be prefetched from ghost nodes based on priority scores includes: sorting ghost nodes from largest to smallest according to priority scores, and selecting the top N ghost nodes as target ghost nodes to be prefetched from the sorting results.

[0048] In practice, the scanned ghost nodes are sorted in descending order of their scores, with the top-ranked nodes considered critical nodes to be reused. The top N nodes of the list are used to generate a prefetch task queue. This greedy strategy ensures that, with limited I / O bandwidth and memory resources, data that contributes the most to performance improvement is prioritized for recovery.

[0049] As a preferred implementation, the process of sorting ghost nodes from largest to smallest according to priority score further includes: when there are two or more ghost nodes with the same priority score, sorting them from smallest to largest according to their depth in the prefix cache tree.

[0050] In practice, a secondary rule is added to the comparison logic of the prefetch sorting. If two ghost nodes C and D have the same priority score, their depths are compared. The node with the smaller depth will be ranked higher and given priority in being selected for the prefetch list. This is because a smaller depth means the node is closer to the root, represents a shorter text prefix, and is more likely to be reused in more different contexts.

[0051] It is evident that in resource-constrained prefetching scenarios, prioritizing the migration of highly versatile shallow data can achieve the greatest improvement in cache hit rate with minimal data transfer.

[0052] As a possible implementation, it also includes: when a data access request does not hit the accelerator video memory and processor memory, reading the target data cache block corresponding to the data access request from the compute fast link extended memory and loading it into the accelerator video memory or processor memory.

[0053] In practical implementation, as a fallback strategy to the aforementioned proactive prefetching mechanism, when an inference request arrives, the system first searches for the required data in the accelerator memory and processor memory. If the search fails (i.e., a miss), and the data exists in a ghost node (indicating it was previously evicted) or has never been loaded, a read operation is immediately initiated, directly loading the target data block from CXL memory into high-speed memory to meet the urgent needs of the current computation. After loading is complete, the page table mapping is updated, and the computation process is restarted.

[0054] The cached data scheduling method provided in this invention introduces a multi-dimensional priority scoring mechanism based on prefix cache tree depth, access frequency, and proximity. Specifically, when the accelerator memory or processor memory occupancy reaches a threshold, not only is the hotness of the data considered, but also the depth of the data in the prefix cache tree is incorporated into the scoring system. Since the prefix cache tree reflects the contextual dependencies between data blocks, shallower nodes usually represent more basic shared prefixes. By comprehensively considering these three indicators to determine the priority score and select the target node to be evicted, it is possible to identify and retain key nodes that, although not frequently accessed in the short term, are crucial to maintaining the integrity of the dependency chain, thus avoiding dependency chain breaks caused by the accidental deletion of key nodes. Therefore, this invention significantly reduces the probability of recomputation caused by cache misses, improves the cache hit rate, and thereby improves the overall computational efficiency and resource utilization of large model inference. Furthermore, when evicting a target node, this invention does not simply discard all information, but instead removes its physical address mapping, marks the target node as a ghost node, and retains the ghost node's metadata in the processor memory, enabling the system to maintain global awareness of evicted data with extremely low memory overhead. By periodically scanning these ghost nodes and recalculating priority scores based on access frequency, recentity, and tree depth, high-value target ghost nodes most likely to be accessed again can be accurately identified from massive amounts of evicted data. Subsequently, this data is proactively read from the extended memory of the fast computation link and loaded into the processor memory. This proactive prefetching mechanism based on ghost node metadata transforms the originally passive, blocking on-demand loading into an active, non-blocking data preparation process. Therefore, this embodiment of the invention effectively masks the high latency of reading data from extended memory, reduces the time inference requests wait for data loading, significantly reduces first-word latency, and ensures the response speed and stability of long text inference services.

[0055] This invention discloses a data caching method. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically: see... Figure 2 The flowchart illustrates another method for caching data according to an exemplary embodiment.

[0056] S201: When the inference task generates a new data cache block, the new data cache block is stored in the accelerator video memory; it is determined whether the hit count of the new data cache block has reached a preset threshold; if so, a data copy operation is performed to copy the new data cache block from the accelerator video memory to the processor memory and the compute fast link extended memory through a copy stream that is different from the computation stream in which the inference task is located; wherein, before the data copy operation is completed, the node corresponding to the new data cache block is locked through a reference locking mechanism, and when the completion of the data copy operation is detected, the reference lock on the node corresponding to the new data cache block is released.

[0057] In this context, the inference task refers to the computational process by which a pre-trained language model generates response text based on input prompts. The copy stream is a dedicated asynchronous data transfer channel in the GPU driver, independent of the inference computation stream. It is specifically used for memory copying between the host and the device, avoiding blocking core matrix operations. Reference locking is a concurrency control mechanism that sets a lock flag on a specific node in the memory management structure to prevent that node from being modified, released, or rescheduled by other threads during data migration.

[0058] In this step, the newly generated KV data initially resides in the accelerator's video memory for rapid access during the current inference step. The access count of this data block is monitored in real time. Once the count reaches a preset threshold, indicating high reuse potential, an independent copy stream is initiated to launch an asynchronous copy task. Crucially, the node is locked at the moment the copy begins, ensuring that the data remains static and is not accidentally deleted throughout the entire window of data transfer from video memory to main memory. The node is only unlocked after the copy stream returns a completion signal, allowing it to participate in subsequent eviction scheduling or be overwritten.

[0059] As a preferred implementation, copying the new data cache block from the accelerator video memory to the processor memory includes: determining whether the access frequency of the parent node of the node corresponding to the new data cache block is greater than or equal to a second preset threshold; if so, generating a Huffman table based on the occurrence frequency of tokens under the parent node; using the Huffman table to perform Huffman encoding on the token sequence in the new data cache block to obtain a compressed data cache block, and filling the metadata header of the compressed data cache block with the Huffman table; and writing the filled compressed data cache block into the compute fast link extended memory.

[0060] In this context, the parent node refers to the node that points directly to the current node in the prefix cache tree, representing the prefix context of the current data block.

[0061] In practice, to further conserve CXL memory storage space and reduce I / O transmission time, adaptive compression is performed before writing data to CXL memory. First, it checks if the parent node of the current data block is sufficiently popular, i.e., whether its access frequency is greater than or equal to a second preset threshold. If so, it indicates that the token distribution in this context exhibits significant statistical regularity (some tokens appear with extremely high probabilities). Utilizing this local statistical characteristic, a Huffman table for the parent node context is dynamically constructed. Subsequently, the Huffman table is used to encode the token sequence (or its corresponding quantization index sequence) in the current data block, generating compressed data. Finally, the generated Huffman table is appended as a metadata header before the compressed data and written to CXL memory.

[0062] As can be seen, this implementation utilizes context-dependent dynamic Huffman coding to achieve high-compression data storage. Since the Huffman table is generated based on high-frequency parent nodes, it is highly targeted, and its compression efficiency is far superior to general dictionary compression. This not only reduces the cost pressure of expanding memory but also improves I / O throughput efficiency during data readback.

[0063] As a feasible implementation method, a Huffman table is generated based on the occurrence frequency of tokens under the parent node, including: obtaining the occurrence frequency of tokens under the parent node, calculating the conditional probability of the token under the parent node based on the occurrence frequency of the token, and determining the encoding length of the token in the Huffman encoding based on the conditional probability; wherein, the encoding length of the token is negatively correlated with the occurrence frequency of the token under the parent node.

[0064] Conditional probability refers to the probability of a specific token appearing given the context of the parent node.

[0065] In practice, the historical logs or counters are traversed to count the frequency of all child tokens under a specific parent node. The frequencies are normalized to obtain a conditional probability distribution. Next, the Huffman tree construction algorithm is executed: the two nodes with the lowest probabilities are merged, and this process is repeated until the root node is formed. The path from the root node to each leaf node is the token's encoding. Because the construction process always prioritizes merging low-frequency items, high-frequency items are naturally closer to the root node, resulting in shorter encodings.

[0066] As can be seen, by accurately calculating conditional probabilities and constructing optimal prefix codes, this implementation method ensures maximum compression efficiency in a specific context. This adaptive mechanism can keenly capture the differences in token distribution under different application scenarios (such as code, dialogue, and translation) and dynamically adjust the compression strategy.

[0067] As a feasible implementation method, tokens Encoding length satisfy: , For tokens In the parent node Frequency of occurrence of the following For tokens In the parent node The frequency of occurrence of the following.

[0068] In practical implementation, the global prefix tree (Radix Tree) is defined as follows: ,in For a set of nodes, Let this be the set of edges. For any KV Cache data block... , its in Each node corresponds to a unique parent node (context). .set up The data in the sequence is quantized into discrete symbol sequences. Composition, in which (This refers to the quantization codebook space, for example, 4-bit quantization) Traditional global compression methods assume that symbols... Follows global distribution Its theoretical average code length lower limit is determined by the global information entropy: Utilizing the contextual relevance of the prefix tree, for a specific node The subsequent distribution of tokens (i.e., KV Cache content) is not random, but follows a specific conditional probability distribution. According to the principles of information theory, the conditional entropy after introducing contextual information... It must be less than or equal to the global entropy: The goal of Huffman compression is to construct a set of local coding tables such that the actual average code length is reduced. Approaching this lower conditional entropy lower bound. The system periodically maintains a frequency matrix. , of which elements Symbols At the node The frequency of occurrence. For hotspot paths, construct a dedicated Huffman tree. For symbols Its encoding length Satisfying the Huffman optimality condition: .

[0069] Define the original data bit width as (For example, FP16 is 16 bits). Under context-aware compression, nodes... Expected compression ratio for: Here, the numerator represents a single value (Symbol) in the original data with a fixed bit width. For CXL interconnects, let the physical bandwidth be... Effective bandwidth after implementing this compression strategy. It can be modeled as: In typical large-scale model inference scenarios, due to the long-tail effect of token distribution, the local entropy of hotspot paths is extremely low. Calculations show that the average compression ratio... This can be 1.5 to 2.5 times faster. This means that with the physical bandwidth remaining unchanged, the data transfer speed (prefetch speed) is increased by 1.5 to 2.5 times, thereby significantly reducing the first-word latency (TTFT) caused by I / O.

[0070] The writing process can be described as follows: ,in, The original data, It is based on the current block's position in the Radix Tree. The indexed Huffman table, The data is compressed; Encode is the encoding operation.

[0071] As a feasible implementation method, after determining whether the access frequency of the parent node of the node corresponding to the new data cache block is greater than or equal to the second preset threshold, the method further includes: if the access frequency of the parent node of the node corresponding to the new data cache block is less than the second preset threshold, then the token sequence in the new data cache block is compressed based on the preset compression dictionary to obtain a compressed data cache block; and the compressed data cache block is written into the computation fast link extended memory.

[0072] The preset compressed dictionary refers to a pre-trained or statically configured general encoding mapping table that covers the general encoding patterns of common tokens in the language model vocabulary and does not rely on real-time context statistics.

[0073] In practice, when the parent node is accessed infrequently, it indicates that the context is relatively unused. In this case, a general preset dictionary is used to compress the data. Although the compression ratio may not be as high as dynamic Huffman coding, it saves the time spent on statistics and table creation, ensuring low latency and consistency in the write process.

[0074] S202: When the occupancy rate of accelerator memory or processor memory reaches the first preset threshold, the priority score of the current storage node is determined based on the access frequency, proximity, and depth in the prefix cache tree of the current storage node of accelerator memory or processor memory; wherein, the prefix cache tree is a tree data structure used to organize the contextual dependencies between data cache blocks, and proximity is used to characterize the proximity of the access time of the current storage node.

[0075] S203: Select the target node to be evicted from the current storage nodes based on the priority score, and check whether the data cache block corresponding to the target node has been stored in the compute fast link extended memory; if so, directly remove the physical address mapping of the target node in the accelerator memory or processor memory; if not, wait for the data cache block to be completely written to the compute fast link extended memory before removing the physical address mapping of the target node in the accelerator memory or processor memory.

[0076] In this step, after selecting the target node to be evicted, the first step is to verify whether the node has completed the asynchronous write process described in S201, i.e., whether it already exists in CXL memory. If the data is already in CXL memory, for example, if it has already been copied there due to a high hit count, or if it has already been evicted once and written, then this copy in high-speed memory is redundant and can be safely unmapped and the space reclaimed directly without any I / O wait, achieving zero-latency eviction. If the data has not yet been written to CXL memory, for example, if it is newly generated hot data that has just been copied but not yet written, or if it is cold data that has never been marked as hot and is about to be evicted, then directly unmapping it will result in permanent data loss. Therefore, the release operation of this node must be paused, and the background operation of writing it to CXL memory must be completed to confirm that the data has been written to disk before unmapping it from high-speed memory.

[0077] S204: Remove the physical address mapping of the target node in the accelerator video memory or processor memory, mark the target node as a ghost node, and retain the ghost node's metadata in the processor memory.

[0078] S205: Periodically scan ghost nodes, determine the priority score of ghost nodes based on their access frequency, proximity, and depth in the prefix cache tree, select target ghost nodes to be prefetched from the ghost nodes based on the priority score, read the data cache block corresponding to the target ghost node from the compute fast link extended memory, and load it into the processor memory.

[0079] As a feasible implementation method, reading the data cache block corresponding to the target ghost node from the computation fast link extended memory and loading it into the processor memory includes: reading the compressed data cache block corresponding to the target ghost node from the computation fast link extended memory; parsing the metadata header of the compressed data cache block to obtain the Huffman table corresponding to the compressed data cache block; performing a decoding operation on the compressed data cache block according to the Huffman table to obtain the original data cache block, and storing the original data cache block into the processor memory.

[0080] In practice, once the prefetching mechanism selects a target ghost node, it initiates a read request from CXL memory. If CXL stores compressed data, the read data stream cannot be used directly. First, the metadata in the packet header is parsed to extract the Huffman table generated during the write operation. Based on the Huffman table, the compressed bitstream is parsed bit by bit to reconstruct the original KV data block. Finally, the reconstructed original data block is stored in a free page of the processor memory, and the page table is updated to make it visible to the inference engine. The read operation can be described as follows: Decode is the decoding operation. Since Huffman decoding is lossless and deterministic, combined with CXL's byte addressing capabilities, streaming decoding can be achieved, i.e.: , The total access time for reading and restoring the original data from CXL memory, Due to the inherent access latency of CXL interconnects, The size of the original data. To achieve effective transmission bandwidth, by maximizing This significantly reduces the transmission time.

[0081] Therefore, this embodiment utilizes a copy stream independent of the computation flow combined with a reference locking mechanism to achieve seamless migration of hot data from accelerator memory to processor memory. This avoids blocking core inference computation during data movement and completely eliminates the risk of data race conditions under concurrent access, ensuring system stability under high load scenarios. Furthermore, an adaptive compression strategy based on the access frequency of the context parent node is introduced. High-frequency shared prefixes are dynamically compressed using Huffman tables, while low-frequency cold data is compressed using a general dictionary. This dual-mode mechanism significantly reduces the memory storage space occupied by the fast computation link expansion and the I / O transmission bandwidth pressure, while preserving the local statistical characteristics of the data, thus significantly increasing the amount of effective information that can be transmitted per unit bandwidth.

[0082] Based on the above embodiments, as a preferred implementation, after selecting the target ghost node to be prefetched from the ghost nodes according to the priority score, the method further includes: obtaining the set of child nodes of the target ghost node in the prefix cache tree, and calculating the historical access transfer probability of each child node in the set of child nodes; predicting the subsequent token sequence most likely to be continuously accessed after the target ghost node is loaded into the processor memory based on the historical access transfer probability; while reading the compressed data cache block corresponding to the target ghost node from the computation fast link extended memory, constructing a pre-decoding lookup table for the subsequent token sequence in parallel using idle computing resources; when the compressed data cache block is read, performing streaming decoding on the compressed data cache block using the pre-decoding lookup table, and writing the decoded original data cache block into fragments into the processor memory until all decoding is completed.

[0083] Predictive access paths refer to the sequence of child nodes most likely to be visited immediately after the current node is visited, inferred from the topology of the prefix cache tree and historical access logs. Pre-decoding lookup tables are pre-built, fast mapping structures optimized for predicted high-frequency subsequent sequences, used to accelerate the decoding process.

[0084] In practical implementation, for the dynamically Huffman compressed data, when a target ghost node (e.g., the node corresponding to the prefix "artificial intelligence") is selected for prefetching, not only is an I / O read request initiated, but the child nodes of that node in the prefix tree (e.g., "artificial intelligence technology," "artificial intelligence application," etc.) are immediately analyzed. By statistically analyzing historical transition probabilities, it can be predicted that the user is highly likely to input the word "technology" next. Based on this prediction, the idle computing power of the CPU is used in the background to pre-simulate and construct an optimized decoding lookup table specifically for the potential path of "artificial intelligence technology." At the same time, data read operations in CXL memory are underway. Once the Huffman table in the header of the data packet is read completely, or the data stream begins to arrive, the decoder does not need to wait for the complete initialization of the general decoding structure, but directly uses the prepared pre-decoding lookup table to perform high-speed matching and decoding of the incoming data stream. Since the predicted high-frequency sequences account for most of the data, this targeted optimization can significantly reduce lookup conflicts and cache misses during decoding, thereby improving decoding speed. The decoded data fragments are directly filled into the processor memory, and the inference engine can start processing after partial data decoding is completed, realizing deep pipeline overlap of I / O, decoding, and computation.

[0085] As can be seen, this implementation guides the pre-allocation of decoding resources by predicting subsequent access paths and uses parallel pipelines to mask the computational and I / O latency of dynamic Huffman decoding. This not only solves the problem of additional computational overhead caused by compressed storage, but also further reduces the first-word latency in long text inference. Especially in the scenario of using dynamic encoding with high compression ratio, it significantly improves the end-to-end response speed of the system.

[0086] The following describes an application embodiment of the present invention, the overall structure of which is as follows: Figure 3As shown, the overall structure is divided into an application layer, an adaptive hierarchical data cache scheduling framework, and a physical memory hierarchy from top to bottom. The topmost application layer is the pre-trained language model inference engine and scheduler, which receives long context requests and passes them to the request processors below. The adaptive hierarchical data cache scheduling framework contains three core modules: a non-blocking hierarchical data placement module manages asynchronous backups across memory hierarchies; a topology-aware priority analyzer calculates priority scores using prefix tree metrics; and an adaptive scheduling mechanism performs memory pressure-aware eviction operations and periodic prefetch instructions. The adaptive hierarchical data cache scheduling framework sends two types of instructions to the physical memory hierarchy below: eviction instructions (memory pressure awareness) and periodic prefetch instructions. The bottommost physical memory hierarchy shows a three-layer storage structure and their interactions: the first layer is accelerator memory, the second layer is processor memory, and the third layer is extended memory for fast computation links. The data flow between the layers is indicated by arrows: there is a two-way interaction of asynchronous backup and load-back between accelerator memory and processor memory, and accelerator memory can perform eviction operations; similarly, there is a two-way interaction of asynchronous backup and periodic boost between processor memory and compute fast link extension memory, and processor memory can also perform eviction operations. The entire architecture achieves efficient data scheduling among multiple levels of physical memory through non-blocking data placement, priority analysis based on prefix tree topology, and adaptive eviction and prefetching mechanisms.

[0087] To effectively manage the massive state space of long-context pre-trained language models, the traditional radix tree structure is extended to support unified management across heterogeneous storage media. The prefix cache tree structure is as follows: Figure 4 As shown, to bridge the gap between memory-resident data and evicted data, the concept of Ghost Nodes is introduced. Active Nodes (Tier-1 / 2): These nodes store valid physical pointers to data blocks residing in accelerator memory or processor memory. They are responsible for handling immediate inference requests. Ghost Nodes (Tier-3 / Evicted): When a node is logically evicted from GPU and CPU memory through unmapping, its tree node object is retained, but its physical memory handle is released, i.e., `host_value=None` (host value is null) and it is marked as `evicted=True` (evicted is marked as true). Crucially, the prefetch mechanism only scans these Ghost Nodes. This design significantly reduces the search space, allowing the scheduler to focus on eviction boundaries—data that is still relevant recently but has been forced out—rather than blindly scanning the entire backup storage.

[0088] To track historical access patterns of millions of evicted nodes without significant memory overhead, a shadow cache based on Cuckoo Hashing was implemented. This structure is decoupled from the primary radix tree and optimized for high-speed lookups and extreme compactness. A compact hash table implementing Cuckoo Hashing is used. Instead of storing the full 64-bit key, each slot stores a short fingerprint (e.g., 8 bits) derived from the full prefix hash, along with the corresponding value payload. Given a full prefix identifier, derived from the radix tree path (e.g., extra_key, token_tuple), the primary bucket index is calculated through its hash. The spare bucket is obtained by XORing it with the fingerprint hash. The search process checks two candidate buckets and matches fingerprints, achieving constant-time access in practice. Each entry contains a compressed tuple required by the priority analyzer: Each field is quantized to 8 bits. The freshness period decays periodically to approximate timeliness with limited storage. HitCount corresponds to the number of hits, FreshnessEpoch corresponds to the freshness period (data accessed more recently is fresher), and Depth corresponds to the depth of a node in the prefix cache tree. For example, if the depth of our root node is 0, then the depth of its child nodes is 1, the depth of its grandchild nodes is 2, and so on. Even when a ghost node is completely removed from the tree structure under memory pressure, its statistical characteristics remain in the shadow cache. This structure is queried when the system scans for ghost nodes or processes new requests. Once a fingerprint matches, we can immediately retrieve historical hit counts and recency, ensuring that recurring prefixes can quickly recover their priority using historical statistics, resulting in higher stability compared to strategies based solely on timeliness.

[0089] To verify the effectiveness of the proposed adaptive hierarchical data caching scheduling framework, a targeted evaluation was conducted, focusing on system request latency and cache hit rate. The experiment aimed to test two key aspects: (1) the effectiveness of the adaptive scheduling strategy compared to the standard benchmark; and (2) the trade-offs between different write consistency strategies under limited memory bandwidth.

[0090] Experimental Setup: The experiment was conducted on an NVIDIA RTX 4090D GPU equipped with 24GB of VRAM. The system was expanded with 32GB of CXL-based memory (Tier 3), configured as a Transparent Non-Uniform Memory Access (NUMA) node (Node 2), accessed via the CXL 2.0 protocol. Considering the total working set size (WSS) of the workload was approximately 25-30GB, two different host memory (Tier 2) configurations were defined to simulate different cache coverage: Configuration A (Standard Tiering, approximately 70% coverage): 20GB host DRAM. This capacity accommodates approximately 66%-80% of the total KV cache, representing a balanced tiered environment where adaptive strategies effectively distinguish between hot and warm data to minimize CXL access. Configuration B (High Stress, approximately 50% coverage): 15GB host DRAM. By limiting Tier 2 to processing only about half of the total working set, heavy eviction pressure was intentionally introduced. This configuration forces frequent data exchange between the host and CXL, creating an ideal stress testing environment for evaluating the stability of write-through and write-back consistency strategies. In all cases, GPU memory usage is limited to 0.8% to ensure frequent tiered data movement.

[0091] The system was evaluated using three multi-turn dialogue datasets: Loogle: a long-context retrieval benchmark involving complex reasoning on extended documents; ShareGPT: a dataset containing frequently shared system cues, testing the ability to retain a large number of prefixes in the upper-level cache; and UltraChat: a long-context instruction dataset with progressively increasing cue lengths (up to approximately 20k tokens), placing significant strain on capacity.

[0092] Comparison 1: Adaptive Scheduling Efficiency (Configuration A): In the first set of experiments, Configuration A (20GB host DRAM) was used to evaluate the core efficiency of the adaptive tiering strategy relative to the standard HiCache Least Recently Used (LRU) benchmark. Figure 5As shown, cache hit rate: On average, this strategy improved cache hit rate by 10.2% across the three datasets. Notably, on ShareGPT, which has frequent system cues, the hit rate reached 0.753, while Least Recently Used (LRU) was 0.627. Response latency (Time to first token generation (TTFT): The improved hit rate translates to significantly faster response times. This method reduced the average TTFT by 30.9% (average reduction: 2.07 seconds → 1.43 seconds). More critical to user experience, tail latency (P90 TTFT) was reduced by an average of 31.8%, with the most significant decrease in Loogle (5.63 seconds → 3.58 seconds). End-to-end latency: Overall system efficiency is reflected in average latency, with a uniform reduction of 20.2% across all workloads, indicating that CXL-aware prefetching effectively masks data retrieval overhead.

[0093] Comparison 2: Impact of Write Consistency Policy (Configuration B): In the second set of experiments, memory constraints are tightened (Configuration B: 15GB host DRAM), and the stability of the write-through policy relative to write-back and Least Recently Used (LRU) benchmarks is evaluated. Figure 6 As shown, while the write-back strategy attempts to reduce bus traffic by limiting writes to only during evicting, it suffers from significant latency spikes. Compared to write-back, this method reduces the average first token generation time (TTFT) by 46.9% (2.71 seconds → 1.44 seconds). Crucially, write-back significantly reduces the P90 TTFT (average 9.24 seconds), while this method maintains a stable 3.57 seconds, resulting in a substantial 61.4% reduction in tail latency. Even under stricter memory constraints, this method maintains a high average hit rate (approximately 0.66), with an average latency reduction of 37.2% (4.36 seconds → 2.74 seconds) compared to the write-back configuration.

[0094] These results confirm that while write-back saves bandwidth during idle periods, it introduces a "stop-and-wait" cost during burst evictions. In contrast, the prefetch-aware write-through strategy smooths out interconnect utilization and provides stable performance even under conditions of severely limited Level 2 memory.

[0095] The following describes a data caching device provided by an embodiment of the present invention. The data caching device described below can be referred to in conjunction with the data caching method described above. See also... Figure 7 The diagram illustrates a structure of a data caching device according to an exemplary embodiment.

[0096] The determination module 100 is used to determine the priority score of the current storage node based on the access frequency, proximity, and depth in the prefix cache tree when the occupancy rate of the accelerator memory or processor memory reaches a first preset threshold. The prefix cache tree is a tree data structure used to organize the contextual dependencies between data cache blocks, and proximity is used to characterize the proximity of the access time of the current storage node.

[0097] The eviction module 200 is used to select the target node to be evicted from the current storage nodes according to the priority score, remove the physical address mapping of the target node in the accelerator video memory or processor memory, mark the target node as a ghost node, and retain the metadata of the ghost node in the processor memory.

[0098] The prefetch module 300 is used to periodically scan ghost nodes. It determines the priority score of ghost nodes based on their access frequency, proximity, and depth in the prefix cache tree. Based on the priority score, it selects target ghost nodes to be prefetched from the ghost nodes, reads the data cache block corresponding to the target ghost node from the compute fast link extended memory, and loads it into the processor memory.

[0099] The cache data scheduling device provided in this invention introduces a multi-dimensional priority scoring mechanism based on prefix cache tree depth, access frequency, and proximity. Specifically, when the accelerator memory or processor memory occupancy reaches a threshold, it considers not only the hotness or coldness of the data but also the depth of the data in the prefix cache tree in the scoring system. Since the prefix cache tree reflects the contextual dependencies between data blocks, shallower nodes usually represent more basic shared prefixes. By comprehensively considering these three indicators to determine the priority score and select the target node to be evicted, it is possible to identify and retain key nodes that, although not frequently accessed in the short term, are crucial to maintaining the integrity of the dependency chain, thus avoiding dependency chain breaks caused by the accidental deletion of key nodes. Therefore, this invention significantly reduces the probability of recomputation caused by cache misses, improves the cache hit rate, and thereby improves the overall computational efficiency and resource utilization of large model inference. Furthermore, when evicting a target node, this invention does not simply discard all information but removes its physical address mapping, marks the target node as a ghost node, and retains the ghost node's metadata in the processor memory, enabling the system to maintain global awareness of evicted data with extremely low memory overhead. By periodically scanning these ghost nodes and recalculating priority scores based on access frequency, recentity, and tree depth, high-value target ghost nodes most likely to be accessed again can be accurately identified from massive amounts of evicted data. Subsequently, this data is proactively read from the extended memory of the fast computation link and loaded into the processor memory. This proactive prefetching mechanism based on ghost node metadata transforms the originally passive, blocking on-demand loading into an active, non-blocking data preparation process. Therefore, this embodiment of the invention effectively masks the high latency of reading data from extended memory, reduces the time inference requests wait for data loading, significantly reduces first-word latency, and ensures the response speed and stability of long text inference services.

[0100] Based on the above embodiments, as a preferred embodiment, it further includes: a checking module, used to check whether the data cache block corresponding to the target node has been stored in the compute fast link extended memory; if so, the step of removing the physical address mapping of the target node in the accelerator video memory or processor memory is directly executed; if not, the step of removing the physical address mapping of the target node in the accelerator video memory or processor memory is executed after the data cache block is completely written to the compute fast link extended memory.

[0101] Based on the above embodiments, as a preferred implementation, it further includes: a storage module, used to store the new data cache block in the accelerator video memory when the inference task generates a new data cache block; a judgment module, used to determine whether the hit count of the new data cache block has reached a preset threshold; if so, the workflow of the copy module is started; the copy module, used to perform a data copy operation to copy the new data cache block from the accelerator video memory to the processor memory through a copy stream different from the computation stream in which the inference task is located; wherein, before the data copy operation is completed, the node corresponding to the new data cache block is locked through a reference locking mechanism, and when the completion of the data copy operation is detected, the reference lock on the node corresponding to the new data cache block is released.

[0102] Based on the above embodiments, as a preferred implementation, the copy module includes: a judgment unit, used to judge whether the access frequency of the parent node of the node corresponding to the new data cache block is greater than or equal to a second preset threshold; if so, the workflow of the generation unit is started; the generation unit is used to generate a Huffman table based on the occurrence frequency of tokens under the parent node; a first compression unit is used to perform Huffman encoding on the token sequence in the new data cache block using the Huffman table to obtain a compressed data cache block, and fill the Huffman table in the metadata header of the compressed data cache block; and a first writing unit is used to write the filled compressed data cache block into the computation fast link extended memory.

[0103] Based on the above embodiments, as a preferred implementation, the generation unit is specifically used for: obtaining the occurrence frequency of the token under the parent node, calculating the conditional probability of the token under the parent node based on the occurrence frequency of the token, and determining the encoding length of the token in Huffman coding based on the conditional probability; wherein, the encoding length of the token is negatively correlated with the occurrence frequency of the token under the parent node.

[0104] Based on the above embodiments, as a preferred implementation method, the token Encoding length satisfy: , For tokens In the parent node Frequency of occurrence of the following For tokens In the parent node The frequency of occurrence of the following.

[0105] Based on the above embodiments, as a preferred implementation, the prefetch module includes: a reading unit, used to read the compressed data cache block corresponding to the target ghost node from the compute fast link extended memory; a parsing unit, used to parse the metadata header of the compressed data cache block and obtain the Huffman table corresponding to the compressed data cache block; and a decoding unit, used to perform a decoding operation on the compressed data cache block according to the Huffman table to obtain the original data cache block and store the original data cache block in the processor memory.

[0106] Based on the above embodiments, as a preferred embodiment, the copy module further includes: a second compression unit, used to compress the token sequence in the new data cache block based on a preset compression dictionary to obtain a compressed data cache block when the access frequency of the parent node of the node corresponding to the new data cache block is less than a second preset threshold; and a second writing unit, used to write the compressed data cache block into the computation fast link extended memory.

[0107] Based on the above embodiments, as a preferred implementation, the eviction module is specifically used to: sort the current storage nodes from smallest to largest according to priority scores, and select the top M current storage nodes as target nodes to be evicted from the sorting results; correspondingly, the prefetch module is specifically used to: sort the ghost nodes from largest to smallest according to priority scores, and select the top N ghost nodes as target ghost nodes to be prefetched from the sorting results.

[0108] Based on the above embodiments, as a preferred implementation, the eviction module is further configured to: sort the current storage nodes in descending order of their depth in the prefix cache tree when two or more current storage nodes have the same priority score; correspondingly, the prefetch module is further configured to: sort the ghost nodes in ascending order of their depth in the prefix cache tree when two or more ghost nodes have the same priority score.

[0109] Based on the above embodiments, as a preferred implementation, the priority scoring of node n... The calculation formula is: ;in, Let n be the frequency of node n's visits. Let n be the proximity of node n. Let n be the depth of node n in the prefix cache tree. , These are the weighting coefficients. The exponential coefficient is used when selecting target nodes to be expelled. A negative value indicates that when selecting the target ghost node to be prefetched... It is a positive value.

[0110] Based on the above embodiments, as a preferred implementation, the prefetching module is specifically used to scan ghost nodes periodically at a preset time interval or a preset number of request processing.

[0111] Based on the above embodiments, as a preferred embodiment, it further includes: a reading module, used to read the target data cache block corresponding to the data access request from the compute fast link extended memory and load it into the accelerator memory or processor memory when the data access request does not hit the accelerator memory or processor memory.

[0112] Based on the above embodiments, as a preferred implementation, the expulsion module includes: an acquisition unit for acquiring the complete prefix hash value of the target node; a mapping unit for mapping the complete prefix hash value to a short fingerprint; wherein the number of bits in the short fingerprint is less than the number of bits in the complete prefix hash value; an encoding unit for encoding the access frequency, recentity, and depth in the prefix cache tree of the target node into a compressed load; a storage unit for storing the short fingerprint and compressed load together in a shadow cache based on a cuckoo hash structure; and a marking unit for marking the target node as a ghost node.

[0113] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0114] Embodiments of the present invention also provide an electronic device, Figure 8 This is a structural diagram of an electronic device according to an exemplary embodiment, such as... Figure 8 As shown, the electronic device includes: a communication interface 1, capable of exchanging information with other devices such as network devices; and a processor 2, connected to the communication interface 1 to enable information exchange with other devices and, when running a computer program, to execute the cached data method provided by one or more of the above-mentioned technical solutions. The computer program is stored on a memory 3.

[0115] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general will label all buses as Bus System 4.

[0116] In this embodiment of the invention, memory 3 is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0117] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 3 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0118] The methods disclosed in the above embodiments of the present invention can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0119] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of the present invention. For the sake of brevity, these will not be described in detail here.

[0120] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described embodiments of the data caching method at runtime.

[0121] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0122] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by processor 2, implements the steps in any of the above-described embodiments of the data caching method.

[0123] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by processor 2, implements the steps in any of the above-described cached data method embodiments.

[0124] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] The foregoing has provided a detailed description of a data caching system, method, apparatus, device, medium, and product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for scheduling data for caching, the method comprising: Applied to a storage architecture that includes accelerator memory, processor memory, and compute fast link extended memory, the method includes: When the occupancy rate of the accelerator memory or the processor memory reaches a first preset threshold, the priority score of the current storage node is determined based on the access frequency, proximity, and depth in the prefix cache tree of the current storage node of the accelerator memory or the processor memory; wherein, the prefix cache tree is a tree data structure used to organize the contextual dependencies between data cache blocks, and the proximity is used to characterize the proximity of the access time of the current storage node; Based on the priority score, a target node to be evicted is selected from the current storage nodes, the physical address mapping of the target node in the accelerator video memory or the processor memory is removed, the target node is marked as a ghost node, and the metadata of the ghost node is retained in the processor memory. The ghost nodes are periodically scanned, and their priority scores are determined based on their access frequency, proximity, and depth in the prefix cache tree. Target ghost nodes to be prefetched are selected from the ghost nodes based on their priority scores, and the data cache block corresponding to the target ghost node is read from the compute fast link extended memory and loaded into the processor memory.

2. The method of claim 1, wherein, Before unmapping the physical address of the target node in the accelerator memory or the processor memory, the process further includes: Check whether the data cache block corresponding to the target node has been stored in the computation fast link extended memory; If so, then directly execute the step of removing the physical address mapping of the target node in the accelerator video memory or the processor memory; If not, then after the data cache block is completely written to the compute fast link extended memory, the step of unmapping the physical address of the target node in the accelerator video memory or the processor memory is executed.

3. The method of claim 1, wherein, The method further includes: When the inference task generates a new data cache block, the new data cache block is stored in the accelerator video memory; Determine whether the number of hits on the new data cache block has reached a preset threshold; If so, then by using a copy stream distinct from the computation stream in which the inference task resides, a data copy operation is performed to copy the new data cache block from the accelerator video memory to the processor memory and the computation fast link extended memory; Specifically, before the data copy operation is completed, the node corresponding to the new data cache block is locked through a reference locking mechanism. When the completion of the data copy operation is detected, the reference lock on the node corresponding to the new data cache block is released.

4. The method of claim 3, wherein, Copying the new data cache block from the accelerator video memory to the processor memory includes: Determine whether the access frequency of the parent node of the node corresponding to the new data cache block is greater than or equal to a second preset threshold; If so, a Huffman table is generated based on the frequency of token occurrence under the parent node; The token sequence in the new data cache block is Huffman encoded using the Huffman table to obtain a compressed data cache block, and the Huffman table is filled into the metadata header of the compressed data cache block. The filled compressed data cache block is written to the compute fast link extended memory.

5. The method of claim 4, wherein, Generate a Huffman table based on the frequency of token occurrences under the parent node, including: Obtain the occurrence frequency of the token under the parent node, and calculate the conditional probability of the token under the parent node based on the occurrence frequency of the token; The encoding length of the token in Huffman coding is determined based on the conditional probability; wherein the encoding length of the token is negatively correlated with the frequency of occurrence of the token under the parent node.

6. The cached data scheduling method according to claim 5, characterized in that, Token Encoding length satisfy: , For tokens In the parent node Frequency of occurrence of the following For tokens In the parent node The frequency of occurrence of the token is given by k, where k is the token index.

7. The method of claim 4, wherein, Reading the data cache block corresponding to the target ghost node from the compute fast link extended memory and loading it into the processor memory includes: Read the compressed data cache block corresponding to the target ghost node from the computational fast link extended memory; Parse the metadata header of the compressed data cache block to obtain the Huffman table corresponding to the compressed data cache block; The original data cache block is obtained by performing a decoding operation on the compressed data cache block according to the Huffman table, and the original data cache block is stored in the processor memory.

8. The method of claim 4, wherein, After determining whether the access frequency of the parent node of the node corresponding to the new data cache block is greater than or equal to the second preset threshold, the method further includes: If the access frequency of the parent node of the node corresponding to the new data cache block is less than the second preset threshold, then the token sequence in the new data cache block is compressed based on the preset compression dictionary to obtain the compressed data cache block. The compressed data cache block is written into the computation fast link extended memory.

9. The method of claim 1, wherein, Based on the priority score, a target node to be evicted is selected from the current storage nodes, including: The current storage nodes are sorted from smallest to largest according to the priority score, and the top M current storage nodes are selected as the target nodes to be evicted from the sorting results. Accordingly, selecting target ghost nodes to be prefetched from the ghost nodes based on the priority score includes: The ghost nodes are sorted from largest to smallest according to the priority score, and the top N ghost nodes in the sorting result are selected as the target ghost nodes to be prefetched.

10. The method of claim 9, wherein, The process of sorting the current storage nodes from smallest to largest according to the priority score also includes: When two or more current storage nodes have the same priority score, they are sorted in descending order of depth in the prefix cache tree; Accordingly, the process of sorting the ghost nodes from largest to smallest according to the priority score also includes: When two or more ghost nodes have the same priority score, they are sorted in ascending order of depth in the prefix cache tree.

11. The cached data scheduling method according to claim 9, characterized in that, Priority score of node n The formula for calculating the priority score of node n is: ； in, Let n be the frequency of node n's visits. Let n be the proximity of node n. Let n be the depth of node n in the prefix cache tree. , These are the weighting coefficients. The exponential coefficient is used when selecting target nodes to be expelled. A negative value indicates that when selecting the target ghost node to be prefetched... It is a positive value.

12. The method of claim 1, wherein, The periodic scanning of the ghost nodes includes: The ghost nodes are scanned at preset time intervals or preset request processing numbers.

13. The method of claim 1, wherein, Also includes: When a data access request does not hit the accelerator memory or the processor memory, the target data cache block corresponding to the data access request is read from the compute fast link extended memory and loaded into the accelerator memory or the processor memory.

14. The method of claim 1, wherein, The target node is marked as a ghost node, and the metadata of the ghost node is stored in the processor memory, including: Obtain the complete prefix hash value of the target node; The complete prefix hash value is mapped to a short fingerprint; wherein the number of bits in the short fingerprint is less than the number of bits in the complete prefix hash value; The access frequency, proximity, and depth of the target node in the prefix cache tree are encoded into a compressed load; The short fingerprint is associated with the compressed payload and stored in a shadow cache based on a cuckoo hash structure; The target node is marked as a ghost node.

15. An electronic device, comprising: include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the cached data scheduling method as described in any one of claims 1 to 14.

Images