Data scheduling method and apparatus, processor, electronic device, and storage medium

Abstract

The present disclosure provides data scheduling methods, apparatuses, processors, electronic devices, and storage media. Embodiments include: in response to a processing request entering a waiting queue, determining a first prefix matching state of the processing request; in response to the processing request entering a pending admission state, determining a second prefix matching state of the processing request; based on the first prefix matching state and the second prefix matching state, determining whether a prefix cache recovery condition is satisfied; in response to determining that the prefix cache recovery condition is satisfied, initiating a recovery request to a tertiary storage to recover missing prefix cache data from the tertiary storage to a secondary storage; based on the recovered prefix cache data in the secondary storage, re-performing prefix matching on the processing request to load, according to a third prefix matching state, prefix cache data in the secondary storage that is re-matched with the processing request to a primary storage to admit and execute the processing request. The above method can effectively reduce processing delay.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and specifically to a data scheduling method, apparatus, processor, electronic device, and storage medium. Background Technology

[0002] In large language model inference, the pre-filling stage of input prompts requires calculating attention key-value pairs (KV) cache data (i.e., attention tensor data) for each token. When the prompts are long or multiple inference requests have the same or similar long prefixes, if existing attention key-value pair cache data cannot be effectively reused, a large amount of redundant computation will occur, leading to increased first-token latency, decreased system throughput, and reduced utilization of computing resources such as graphics processing units (GPUs). Therefore, modern inference systems typically attempt to reuse already computed attention key-value pair cache data.

[0003] With the development of long context models and high-concurrency service scenarios, the video memory of a single computing device can no longer accommodate all reusable prefix cache data. Summary of the Invention

[0004] In a multi-tiered storage architecture, when a processing request is queued in a high-concurrency waiting queue awaiting admission, prefix cache data that could have been reused by that request may be evicted or removed from the upper-layer storage due to the execution of other requests (e.g., preceding requests) or data backup pressure, leading to a degradation of the hit (i.e., prefix matching) state. Existing scheduling technologies lack mechanisms to detect and salvage such cache data degradation or matching path (i.e., anchor point) failures that occur during the waiting period before the request is actually admitted. Typically, they can only admit the request based on the degraded shallow prefix matching result and re-perform pre-filling and other computational processing, making it difficult to effectively utilize the large amount of prefix cache data still existing in the lower-layer storage, resulting in serious waste of computing power and increased latency.

[0005] To address the aforementioned issues, this disclosure provides a data scheduling method, apparatus, processor, electronic device, and storage medium for dynamically monitoring cache state changes during the queuing period of processing requests in a multi-tiered storage architecture, and proactively intercepting and recovering recoverable prefix cache data before admission.

[0006] According to one aspect of this disclosure, a cached data scheduling method is provided, applied to a storage architecture including primary storage, secondary storage, and tertiary storage. The method includes: in response to a processing request entering a waiting queue, determining a first prefix matching state of the processing request; in response to the processing request entering a pending admission state, determining a second prefix matching state of the processing request; based on the first and second prefix matching states, determining whether a prefix cache recovery condition is met; in response to determining that the prefix cache recovery condition is met, initiating a recovery request to the tertiary storage to restore the missing prefix cache data from the tertiary storage to the secondary storage; and based on the restored prefix cache data in the secondary storage, re-prefix matching is performed on the processing request to load the re-matched prefix cache data in the secondary storage to the primary storage according to a third prefix matching state, so as to admit and execute the processing request.

[0007] In some embodiments, the first prefix matching state includes: a first-level matching length and a first-level matching length, wherein the first-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the waiting queue, and the first-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the waiting queue; and the second prefix matching state includes: a second-level matching length and a second-level matching length, wherein the second-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the admission-waiting state, and the second-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the admission-waiting state.

[0008] In some embodiments, the first prefix matching state further includes the sum of the first level matching length and the first level matching length, and the second prefix matching state further includes the sum of the second level matching length and the second level matching length.

[0009] In some embodiments, determining whether the prefix cache recovery condition is met based on the first prefix matching state and the second prefix matching state includes: determining that the prefix cache recovery condition is met when it is determined that the prefix cache data has degraded or that there is a shallow anchor point recovery opportunity.

[0010] In some embodiments, prefix cache data degradation is determined when the following condition is met: the sum of the first-level matching length and the first-level matching length is greater than the sum of the second-level matching length and the second-level matching length.

[0011] In some embodiments, the conditions for determining that the prefix cache data has degraded further include: the length of the first secondary match is greater than zero; and the length of the second secondary match is zero.

[0012] In some embodiments, when it is determined that the prefix cache data has not degraded, a determination is made regarding whether there is an opportunity to recover from shallow anchors.

[0013] In some embodiments, a shallow anchor point recovery opportunity is determined when the following conditions are met: the second-level matching length is less than or equal to a preset shallow prefix threshold; and the remaining unmatched length of the processing request is greater than or equal to a preset recovery trigger threshold, wherein the remaining unmatched length of the processing request is the total length of the processing request minus the sum of the second-level matching length and the second-level matching length.

[0014] In some embodiments, the conditions for determining the existence of a shallow anchor point recovery opportunity further include: the second secondary matching length is zero; and the second primary matching length is greater than zero.

[0015] In some embodiments, initiating a recovery request to the tertiary storage to recover the missing prefix cache data from the tertiary storage to the secondary storage includes: pausing the admission processing request; determining the upper limit of the recovery range of the prefix cache data from the tertiary storage; initiating a restricted prefetch request to the tertiary storage based on the second prefix matching status and the upper limit of the recovery range; and writing the missing prefix cache data obtained through the restricted prefetch request to the secondary storage, wherein the data length of the missing prefix cache data is limited by the upper limit of the recovery range.

[0016] In some embodiments, initiating a restricted prefetch request to the tertiary storage based on the second prefix matching state and the upper limit of the recovery range includes: determining the end node corresponding to the matching length in the second prefix matching state as the prefix matching anchor point; and initiating a restricted prefetch request to the tertiary storage based on the upper limit of the recovery range, starting from the prefix matching anchor point.

[0017] In some embodiments, determining the upper limit of the recovery range from the prefix cache data in the tertiary storage includes: in response to determining that the prefix cache data has degraded, determining the upper limit of the recovery range as the smaller value between the total length of the processing request and the sum of the first-level matching length and the first-level matching length; and in response to determining that there is a shallow anchor point recovery opportunity, determining the upper limit of the recovery range as the smaller value between the total length of the processing request and the sum of the second-level matching length and the preset recovery window length.

[0018] In some embodiments, loading the prefix cache data that is rematched with the processing request in the secondary storage to the primary storage according to the third prefix matching state to admit and execute the processing request includes: loading the matched prefix cache data in the secondary storage to the primary storage in response to the third prefix matching state indicating that the matching length of the prefix cache data that matches the processing request in the secondary storage is greater than zero; and admitting the processing request according to the third prefix matching state to perform pre-filling processing on the remaining unmatched portion of the processing request.

[0019] In some embodiments, the method further includes: in response to a third prefix matching state indicating that the matching length of the prefix cache data matching the processing request in the secondary storage is zero, admitting the processing request according to a second prefix matching state to perform pre-filling processing on the remaining unmatched portion of the processing request.

[0020] In some embodiments, the method further includes: in response to determining that the prefix cache recovery condition is not met, performing pre-filling processing on the remaining unmatched portion of the processing request based on the second prefix matching state admission processing request.

[0021] In some embodiments, admitting the processing request includes: transferring the processing request from a waiting queue to an execution state; and allocating computing resources located in primary storage to the processing request so that the processing request performs pre-filling processing.

[0022] In some embodiments, the processing request includes an inference request for a large language model, and the prefix cache data includes attention key-value pair cache data, wherein the attention key-value pair cache data is attention tensor data calculated for the prompt words of the inference request during the pre-filling process of the large language model.

[0023] According to another aspect of this disclosure, a data scheduling apparatus is provided, comprising: a first matching module, configured to determine a first prefix matching state of a processing request in response to entering a waiting queue; a second matching module, configured to determine a second prefix matching state of a processing request in response to entering a pending admission state; a recoverable detection module, configured to determine whether a prefix cache recovery condition is met based on the first and second prefix matching states; a recovery module, configured to suspend the admission processing request and initiate a recovery request to tertiary storage in response to determining that the prefix cache recovery condition is met, so as to restore the missing prefix cache data from the tertiary storage to the secondary storage; and an execution module, configured to re-match the prefix of the processing request based on the restored prefix cache data in the secondary storage, so as to load the re-matched prefix cache data in the secondary storage to the primary storage according to a third prefix matching state, so as to admit and execute the processing request.

[0024] According to another aspect of this disclosure, a processor is provided, comprising: a first memory for storing ready prefix cache data that can directly participate in model computation; a second memory for storing recoverable prefix cache data that can be transferred to the first memory by loading; a third memory for storing underlying prefix cache data that is pulled to the second memory by a restricted prefetch request; and an execution unit coupled to the first memory, the second memory, and the third memory, the execution unit being configured to execute the cache data scheduling method described above in response to receiving a processing request.

[0025] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory that non-transitory stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the cached data scheduling method described above.

[0026] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the cached data scheduling method described above.

[0027] The cached data scheduling method provided by the embodiments of this disclosure can promptly detect the loss of prefix cached data caused by other requests crowding out the queue during the queuing period by comparing the cache matching differences between the processing request when it enters the waiting queue (first prefix matching state) and when it is about to be admitted for execution (second prefix matching state). Once the recovery conditions are met, the system will proactively load the missing prefix cached data from the underlying third-level storage back to the second-level storage before admission execution, and continue to execute the request after rematching. This significantly improves the prefix cache hit depth of the processing request at the final execution, increases the reuse efficiency of existing prefix cached data, avoids unnecessary repeated pre-filling calculations caused by cached data degradation or temporary cache path failure, and thus effectively reduces processing latency and significantly improves the utilization of computing resources and the stability of request processing. Attached Figure Description

[0028] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0029] Figure 1 An exemplary flowchart of a data scheduling method according to an embodiment of the present disclosure is shown.

[0030] Figure 2 An exemplary flowchart of a method for restoring prefix cache data from tertiary storage to secondary storage according to an embodiment of the present disclosure is shown.

[0031] Figure 3 An exemplary state machine diagram of a pre-admission recovery state transition for a processing request is shown according to an embodiment of the present disclosure.

[0032] Figure 4 An interactive sequence diagram of pre-admission recovery of prefix cache data according to an embodiment of the present disclosure is shown.

[0033] Figure 5 An exemplary block diagram of a data scheduling apparatus according to an embodiment of the present disclosure is shown.

[0034] Figure 6 An exemplary block diagram of a processor according to an embodiment of the present disclosure is shown.

[0035] Figure 7 An exemplary block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0036] Figure 8 A block diagram of an example computer device according to an embodiment of the present disclosure is shown.

[0037] Figure 9 An exemplary block diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0040] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0041] In the description of this disclosure, "prefix cache data" includes, but is not limited to, attention key-value (KV) cache data (i.e., attention tensor data) used for reusing computation results in Large Language Model (LLM) inference. "Processing request" includes, but is not limited to, inference requests in a Large Language Model inference service. "Pre-filling processing" includes, but is not limited to, the initial attention computation process on the input prompt sequence in the processing request. "Admission" refers to the process by which the scheduler allows a processing request to enter, for example, a model computation engine, to begin actual execution.

[0042] In existing large language model inference systems, to reduce the overhead of long prompt words in the pre-filling stage, a prefix cache reuse mechanism based on a prefix tree (or Radix Tree) is typically used. The system organizes the token prefixes of historical processing requests into a tree-like index structure, where each node corresponds to a token prefix segment and its associated prefix cache data. When a new processing request arrives, the system searches for the longest common prefix in the prefix tree along the token sequence, thus reusing the prefix cache data of the matching portion. However, due to the limited capacity of primary storage (such as GPU memory), the system typically employs a multi-tiered storage architecture for hierarchical management. Specifically, temporarily inactive but potentially reusable prefix cache data is moved from primary storage to secondary storage (such as host DRAM); to further expand reuse capacity, the system also stores data in tertiary storage, such as a level-3 cache or storage cache. Different storage tiers have different semantics and access costs: data in Tier 1 storage is ready prefix cache data, which can be directly used in model calculations; data in Tier 2 storage is recoverable prefix cache data, which usually needs to be transferred to Tier 1 storage through a load-back operation before it can be used; and data in Tier 3 storage needs to be loaded to Tier 2 storage through a prefetch operation before it can be further recovered and used.

[0043] In the aforementioned multi-tiered storage architecture, requests are not immediately executed after entering the system's waiting queue. The queuing time depends on various factors, including system concurrency, the processing time of preceding requests, and memory availability. Since the cache state may change during the waiting period, an Admission Rematch operation can be performed before the request is actually admitted to reflect the latest upper-level cache state seen by the request at the admission time. However, this approach still passively relies on what can be matched in the current prefix tree and cannot address the cache eviction degradation problem caused by requests waiting in the queue.

[0044] Specifically, the prefix matching state seen by a processing request upon enqueuing is merely a momentary snapshot. During the queuing period, preceding processing requests continuously generate new prefix cache data and write it to secondary storage. Due to the limited capacity of secondary storage, this continuous writing triggers a cache eviction mechanism, causing the deletion of deeper matching paths that the current processing request could originally match in secondary storage. Therefore, when the processing request finally reaches the admission phase and undergoes re-matching, its previous deep prefix paths are no longer reachable, and the matching process stops at a shallow node in the prefix tree. In this disclosure, the non-zero shallow prefix node hit in the current prefix match (i.e., the end node of the matching path in the prefix tree) is referred to as a "shallow anchor."

[0045] When faced with this degradation from deep to shallow, the existing rematching mechanism can only return the matching length of the shallow anchor point, while the hit length of the secondary storage will become shorter, or even zero. The existing scheduler interprets this result as "there is no more recoverable prefix cache data," thus allowing the processing request to be admitted directly with a very shallow prefix matching state. As a result, the system will not actively query the lower-level tertiary storage to see if there is still any deep prefix cache data, but will directly hand over the large number of unmatched tokens (i.e., suffixes) remaining in the processing request to the primary storage for re-pre-filling. For long prompt word requests, this admission operation will cause large-scale redundant calculations, resulting in a huge waste of computing resources.

[0046] This demonstrates that existing scheduling mechanisms lack dynamic monitoring of cache hit status staleness and degradation during queuing, and fail to establish an effective interception and recovery loop before admission. In high-concurrency, long-context cold-start or replay scenarios, this deficiency leads to significant phased performance degradation: as cache pressure accumulates, the matching status of many requests deteriorates severely during admission, resulting in long-tail deterioration of the Time To First Token (TTO), and a sharp drop in overall system throughput. Therefore, there is an urgent need for an intelligent scheduling mechanism that can proactively detect degradation status before admission and use shallow anchors for interception control.

[0047] To overcome the shortcomings of the existing technology, this disclosure proposes a dynamic cache data scheduling mechanism based on a multi-layer storage architecture. This mechanism can dynamically compare the prefix matching status of a processing request when it enters the waiting queue with that when it is about to be admitted. Before admission is executed, it can proactively restore the lost missing prefix cache data from the third-level storage to the second-level storage and re-match and load it into the first-level storage. This improves the reuse efficiency of historical prefix cache data, effectively avoids large-scale repetitive calculations caused by cache failure during queuing, significantly reduces processing latency, and improves the overall throughput performance and resource utilization of the system.

[0048] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0049] Figure 1 An exemplary flowchart of a data scheduling method 100 according to an embodiment of the present disclosure is shown.

[0050] In step S102, in response to the processing request entering the waiting queue, the first prefix matching status of the processing request is determined.

[0051] In step S104, in response to the processing request entering the admission state, the second prefix matching state of the processing request is determined.

[0052] In step S106, based on the first prefix matching state and the second prefix matching state, it is determined whether the prefix cache recovery condition is met.

[0053] In step S108, in response to determining that the prefix cache recovery conditions are met, a recovery request is initiated to the tertiary storage to restore the missing prefix cache data from the tertiary storage to the tertiary storage.

[0054] In step S110, based on the restored prefix cache data in the secondary storage, the processing request is re-prefix matched, so that according to the third prefix matching status, the prefix cache data that is re-matched with the processing request in the secondary storage is loaded into the primary storage to admit and execute the processing request.

[0055] Therefore, by comparing the prefix matching state changes when a processing request enters the waiting queue and enters the admission state, this disclosure effectively identifies the opportunity to recover prefix cache data from the lower storage layer, thereby effectively reducing the repeated processing of processing requests and improving the effective utilization and stability of resources.

[0056] As described above, the multi-level storage architecture according to embodiments of this disclosure typically involves a three-level or more storage structure, including a first-level storage, a second-level storage, and a third-level storage.

[0057] In some embodiments, Tier 1 storage can refer to device-level memory that interacts directly with the computing unit, such as High Bandwidth Memory (HBM) of a graphics processing unit (GPU), whose internal prefix cache data (e.g., device KV cache) can directly participate in model computation; Tier 2 storage can refer to host-level memory devices, such as host dynamic random access memory (HDRAM), whose internal prefix cache data (e.g., host KVcache) does not directly participate in computation for the time being, but can be transferred to Tier 1 storage through a fast load-back operation; Tier 3 storage can refer to a lower-level storage system with larger capacity and lower cost but relatively higher access latency, such as local disk devices (e.g., solid-state drives (SSDs), remote storage servers, distributed memory systems, or dedicated Tier 3 cache (L3 Cache) services, etc., whose internal data needs to be pulled to Tier 2 storage through a prefetch operation before it can be further restored and used.

[0058] It should be noted that this disclosure does not strictly limit the specific physical storage types and device forms of the aforementioned storage levels. The core lies in the hierarchical progression between different storage levels in terms of data readiness, access latency, and recovery costs. In some embodiments, the multi-level storage architecture can also be reasonably extended or modified. For example, in some hardware architectures, the first and second levels of storage may physically share a unified memory mechanism, thus achieving a conceptual merging of the first and second levels; or, in ultra-large-scale clusters, the system may contain not only third-level storage but also a fourth-level network storage as a cold backup. The scheduling mechanism described in this disclosure can also be adapted to these derived storage architectures with hierarchical caching characteristics.

[0059] The following will describe specific examples of this disclosure in detail with reference to Table 1.

[0060] Table 1

[0061] Table 1 shows a variable definition table according to embodiments of the present disclosure.

[0062] In step S102, in response to the processing request entering the waiting queue, the first prefix matching status of the processing request is determined.

[0063] In some embodiments, the processing request includes an inference request for a large language model, and the prefix cache data includes attention key-value pair cache data, wherein the attention key-value pair cache data is attention tensor data calculated for the prompt words of the inference request during the pre-filling process of the large language model.

[0064] However, those skilled in the art will understand that the application scenarios for processing requests are not limited to simple text generation, but can also include various high-concurrency model service scenarios with a large amount of context sharing and prefix reuse requirements. In some examples, processing requests may include requests for code generation model services (reusing shared code repository context), processing requests in long document question answering systems (reusing the same reference document fragments), query requests in retrieval augmented generation (RAG) systems, or state inference requests in agent trajectory replay systems, etc. In these scenarios, different requests typically share extremely long prefix sequences, and reusing prefix cache data has significant performance benefits.

[0065] In high-concurrency model service systems, when a new processing request arrives, it cannot be immediately scheduled for execution because the system's primary storage resources (such as GPU batch processing capacity or video memory) may be occupied by other executing requests. Therefore, it is placed in the system's waiting queue. The time a processing request stays in the waiting queue typically depends on various factors, including the current system load, the processing length of preceding requests, the duration of the decoding process, and the resource scheduling strategy employed by the system.

[0066] At the very beginning of a request's entry into the waiting queue, the system performs an initial prefix match (or prefetch-related match probe) to capture a snapshot of the cache state that the request can see at the moment of enqueueing. This snapshot is the first prefix match state, which records the optimal or deeper cache reuse potential that the request has historically observed.

[0067] In some embodiments, the first prefix matching state may include a first-level matching length and a first-level matching length. The first-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the waiting queue, and the first-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the waiting queue.

[0068] In some examples, the first-level match length can be the historically observed first-level cache ready state prefix length shown in Table 1. P_o The first and second level matching lengths can be the historically observed second-level cache recoverable prefix lengths shown in Table 1. H_o .

[0069] Specifically, the matching length represents the number of consecutive tokens that can be matched between the input sequence of the processing request (e.g., the token sequence of the prompt word) and the existing prefix cache data in the storage in the prefix tree (or Radix Tree). The first-level matching length represents the number of tokens that are fully ready in the first-level storage and can be directly used for model calculation at the enqueue time; the first- and second-level matching lengths represent the number of tokens that exist in the second-level storage at the enqueue time and can be transferred to the first-level storage for reuse through the loading operation.

[0070] In some embodiments, in addition to the matching length described above, the first prefix matching state may also include other possible state information to assist in subsequent comparison and judgment. For example, the first prefix matching state may also include the sum of the first-level matching length and the first-level matching length (i.e., the total historically recoverable length shown in Table 1). R_o ), used to characterize the maximum depth of the overall reusable prefix of the request at the enqueue time.

[0071] In some embodiments, the first prefix matching status may further include node identifiers in the prefix tree corresponding to the matching length, timestamp information, and the total length of the complete input prompt words for the request itself.

[0072] In step S104, in response to the processing request entering the admission state, the second prefix matching state of the processing request is determined.

[0073] In some embodiments, when the processing request in the waiting queue finishes queuing and the system's primary storage or computing unit releases sufficient resources, the scheduler selects the processing request to prepare for the execution phase. At this point, the processing request enters the so-called admission state (a critical state before being admitted for execution). Because other preceding requests in the execution state may continuously generate and write new prefix cache data during the queuing period, leading to insufficient upper-level storage (e.g., secondary storage) space and triggering a cache eviction mechanism, some of the prefix cache data observed when the processing request was enqueued may have already been overwritten or cleared.

[0074] To avoid addressing errors or execution exceptions caused by using expired matching results, the system performs a second prefix match against the latest cache tree before processing a request and allowing it to be admitted. The state determined by this second match is the second prefix match state, which reflects the latest upper-level cache state that the request will see at the actual admission time.

[0075] In the embodiments of this disclosure, the matching status of prefix cache data in each level of storage is clearly distinguished.

[0076] In some embodiments, the second prefix matching state includes: a second first-level matching length and a second second-level matching length. The second first-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the admission-waiting state; the second second-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the admission-waiting state.

[0077] In some examples, the second-level matching length corresponds to the pre-admission current-level cache ready state prefix length shown in Table 1. P_c The second-level matching length corresponds to the recoverable prefix length of the current second-level cache before admission, as shown in Table 1. H_c .

[0078] Furthermore, to facilitate comparison with historical states to determine whether cache degradation has occurred, the second prefix matching state may also include the sum of the second-level matching length and the second-level matching length (i.e., the current recoverable total length shown in Table 1). R_c ), used to characterize the overall reusable prefix depth that the processing request can actually directly utilize before admission.

[0079] In step S106, based on the first prefix matching state and the second prefix matching state, it is determined whether the prefix cache recovery condition is met.

[0080] In some embodiments, determining whether the prefix cache recovery condition is met based on the first prefix matching state and the second prefix matching state includes: determining that the prefix cache recovery condition is met when it is determined that the prefix cache data has degraded or that there is a shallow anchor point recovery opportunity.

[0081] Since the system will incur predictable performance loss in both of these situations, the system will trigger an active interception and recovery process whenever either of these situations occurs.

[0082] In some embodiments, prefix cache data degradation is determined when the following condition is met: the sum of the first-level matching length and the first-level matching length is greater than the sum of the second-level matching length and the second-level matching length.

[0083] Specifically, if, during the queuing period, the current total recoverable length that the processing request can see when awaiting admission (e.g., in Table 1) R_c (), less than the total recoverable length of history observed at the time of its enlistment (e.g., in Table 1) R_ o If this is true, it means that some of the deep prefix cache data that originally existed in the upper storage layer has been squeezed out or eliminated by new data generated by other requests.

[0084] For example, when a request first enters the waiting queue, it can match prefix cache data with a depth of 100K words. However, when the request is about to be admitted after queuing, it is found that it can only match data with a depth of 896 words. At this point, it can be determined from the matching status that a clear degradation phenomenon has occurred, where the prefix cache data matching goes from deep to shallow.

[0085] In some examples, the historical recoverable total length and the current recoverable total length can be directly included in the first prefix matching state and the second prefix matching state determined in steps S102 and S104, respectively, so that they can be directly invoked in step S106. In other examples, to save on state record storage overhead, the historical recoverable total length and the current recoverable total length may not be explicitly recorded in steps S102 and S104, but are instead calculated by the system in step S106 by adding the first-level matching length and the first-level matching length in the first prefix matching state in real time, and by adding the second-level matching length and the second-level matching length in the second prefix matching state in real time.

[0086] In some embodiments, in order to more accurately pinpoint degradation caused by secondary storage eviction, the conditions for determining that prefix cache data has degraded further include: a first secondary matching length greater than zero; and a second secondary matching length of zero.

[0087] Specifically, the length of the first and second level matches is greater than zero (e.g. H_o>0 This indicates that the processing request historically had a clearly existing recoverable prefix data located in secondary storage. The second-level match length is zero (e.g., ...). H_c=0 This indicates that the matching path for the request in secondary storage has been completely broken or lost when admission is imminent (e.g., due to capacity limitations resulting in total eviction).

[0088] In a specific example, the logic for determining the degradation of the aforementioned prefix cache data can be mathematically expressed using the following formula 1: D = I(H_o>0 and H_c = 0 and R_o>R_c) (Formula 1) in, D This indicates the degradation determination result. I( ) This is an indicator function (it takes the value 1 when all the conditions in the parentheses are true, otherwise it takes the value 0). H_o Indicates the length of the first and second level matches. H_c Indicates the length of the second-level match. R_o This represents the sum of the first-level matching length and the first- and second-level matching lengths. R_c This represents the sum of the second-level matching length and the second-level matching length. When all three conditions in Formula 1 are met simultaneously (i.e., ...), ... D=1 The system determines that a clear, recoverable prefix cache data degradation has occurred, and thus confirms that the prefix cache recovery conditions are met.

[0089] As a result, the system can avoid misjudging normal matching fluctuations and ensure that recovery operations are only triggered when a secondary storage path failure actually occurs.

[0090] In some embodiments, when it is determined that the prefix cache data has not degraded, a determination is made regarding whether there is an opportunity to recover from shallow anchors.

[0091] For example, if no valid historical second-level storage hit is recorded when the request is enqueued, or if the aforementioned degradation judgment formula does not hold (i.e., D=0), the system will not directly abandon the recovery attempt. Instead, it will further initiate a judgment on whether there is a shallow anchor recovery opportunity. It should be noted that a "shallow anchor" refers to a non-zero shallow prefix node hit during the current prefix tree matching process. In other words, when the matching path of the processing request in the current upper-level storage is short, the node corresponding to the end of that matching path is a shallow anchor. Although this anchor is shallow, it still contains valid contextual features and can serve as a starting point for probing down to the lower-level third-level storage and recovering subsequent prefix cache data.

[0092] In some embodiments, a shallow anchor point recovery opportunity is determined when the following conditions are met: the second-level matching length is less than or equal to a preset shallow prefix threshold; and the remaining unmatched length of the processing request is greater than or equal to a preset recovery trigger threshold, wherein the remaining unmatched length of the processing request is the total length of the processing request minus the sum of the second-level matching length and the second-level matching length.

[0093] Specifically, the remaining unmatched length in processing the request can be the length of tokens in the request that have not yet been matched and need to be recalculated as suffixes. When the second-level matching length (e.g.) P_c Extremely short (i.e., equal to or below the shallow prefix threshold) and the remaining unmatched length (e.g.) L_remWhen the length of a request is extremely long (i.e., reaches or exceeds the trigger threshold), allowing access directly would result in a massive computational burden. In this case, even without explicit records of prefix cache data degradation, the third-level storage is very likely to contain historical deep prefix cache data of this long prompt word request, thus the system can determine that it has significant detection and recovery value.

[0094] In some embodiments, the conditions for determining the existence of a shallow anchor point recovery opportunity further include: the second secondary matching length is zero; and the second primary matching length is greater than zero.

[0095] Specifically, the second and second level matching lengths (e.g.) H_c A value of zero means there is currently no recoverable path in the secondary storage, which is a prerequisite for probing into the tertiary storage. The matching length of the secondary storage (e.g., ...) P_c The condition that the length of the second-level match must be greater than zero can serve as a cold request filter. P_c> 0 The system mandates that currently processed requests must have at least a non-zero device-ready prefix (i.e., not entirely new requests without any matching elements), effectively filtering out completely cold-start requests. This prevents the system from blindly launching low-yield or completely ineffective probes from the root node of the prefix tree to the underlying storage, thus avoiding unnecessary waste of system bus and memory bandwidth.

[0096] In a specific example, the logic for determining the recovery opportunity of the shallow anchor point can be mathematically expressed by the following formula 2: S=I(H_c=0 and 0<P_c≤T_s and L R_c≥T_p) (Formula 2) in, S This indicates the result of the shallow anchor point recovery opportunity assessment. I( ) This is an indicator function. H_c Indicates the length of the second-level match. P_c Indicates the length of the second-level match. T_s This represents the preset shallow prefix threshold. L Indicates the total length of the request to be processed (e.g., the total length of the prompt). R_c This represents the sum of the second-level matching length and the second-to-second-level matching length, therefore L R_c This represents the remaining unmatched length of the request (e.g., in Table 1). L_rem ), T_p This represents the preset recovery trigger threshold. When all conditions in this formula are met simultaneously (i.e., ... S=1The system determines that the current request has a shallow anchor point recovery opportunity, and thus confirms that the prefix cache recovery conditions are met. By combining degradation detection and shallow anchor point detection, this disclosure can comprehensively cover various recalculation crises caused by cache path failure, achieving accurate interception before admission.

[0097] In some embodiments, Formula 2 can be S=I(H_c=0 and 0<P_c<T_s and L R_c>T_p), That is, the second-level matching length is less than the preset shallow prefix threshold and the remaining unmatched length of the processed request is greater than the preset recovery trigger threshold.

[0098] In step S108, in response to determining that the prefix cache recovery conditions are met, a recovery request is initiated to the tertiary storage to restore the missing prefix cache data from the tertiary storage to the tertiary storage.

[0099] In some embodiments, the access control logic for determining whether a recovery request is triggered can be expressed by the following formula 3: G=D∨S (Formula 3) Here, G represents whether pre-admission recovery is triggered. G is true when either D (the degradation determination result) or S (the shallow anchor recovery opportunity determination result) in Formula 3 above is true. This means the system has either detected explicit data loss during queuing or captured a recalculation crisis for a non-zero shallow prefix long request. As long as either of these conditions is met, the system will not allow the processing request to immediately enter the current round of pre-filling computation, but will proactively trigger the recovery process to the underlying storage.

[0100] When the system attempts to proactively fetch missing prefix cache data from the third-level storage at shallow anchor points, if the complete suffixes of the processing request, which may be as long as tens of thousands of words, are prefetched to the second-level storage all at once without restriction, it will not only cause unnecessary input / output (I / O) and memory bandwidth overhead, but also occupy the available space of the second-level storage in a short period of time, trigger more unnecessary cache evictions, and thus drag down the cache hit rate of other requests.

[0101] Therefore, in some embodiments, this disclosure provides an access-controlled and scope-limited recovery mechanism when a recovery request is triggered.

[0102] Figure 2 An exemplary flowchart of a method 200 for restoring prefix cache data from tertiary storage to secondary storage according to an embodiment of the present disclosure is shown.

[0103] In step S202, the admission processing request is suspended.

[0104] Step S202 is the access control mechanism described in the embodiments of this disclosure. When it is confirmed that the recovery conditions are met, the scheduler will actively block the processing request, causing it to remain in the waiting queue or a dedicated recovery queue, preventing it from being allocated computing resources and entering the pre-filling execution phase in its current shallow matching state.

[0105] By pausing admissions, we can ensure that the benefits of recovery operations directly impact the currently processed request. If admissions are granted without interception, the current request will be forced to undergo extensive recalculation in a shallow hit state. Subsequent retrieval or recalculation of the deep cache may indirectly benefit future requests, or it may crowd out the matching prefixes of subsequent requests. In either scenario, the current request may suffer severe latency penalties (i.e., false recovery), increasing the latency of the first metaword. Pausing admissions ensures that the system waits for the underlying data to be available, allowing the current request to execute with optimal efficiency.

[0106] In step S204, the upper limit of the recovery range of the prefix cache data from the tertiary storage is determined.

[0107] To avoid the impact of unrestricted prefetching on secondary storage mentioned above, in some embodiments, initiating a restricted prefetch request to tertiary storage based on the second prefix matching state and the upper limit of the recovery range includes: determining the end node corresponding to the matching length in the second prefix matching state as the prefix matching anchor point; and initiating a restricted prefetch request to tertiary storage based on the upper limit of the recovery range, starting from the prefix matching anchor point.

[0108] In some embodiments, determining the upper limit of the recovery range from the prefix cache data in the tertiary storage includes: in response to determining that the prefix cache data has degraded, determining the upper limit of the recovery range as the smaller value between the total length of the processing request and the sum of the first-level matching length and the first-level matching length; and in response to determining that there is a shallow anchor point recovery opportunity, determining the upper limit of the recovery range as the smaller value between the total length of the processing request and the sum of the second-level matching length and the preset recovery window length.

[0109] Specifically, the sum of the first-level matching length and the first-level matching length can be the total historical recoverable length. R_o The second-level matching length can be the length of the currently ready prefix. P_c The preset recovery window length can be as shown in Table 1. W_f .

[0110] The above-mentioned constraint logic can be mathematically expressed using the following constrained recovery range formula 4: B=min(L, B ′ ) (Formula 4) in, B This is the final upper limit of the recovery range. LThe total length of the requests processed. For condition variables. B′ If the system's recovery is triggered by degradation of prefix cached data, then B′=R_o (That is, only the maximum depth observed when the swarm was joined was restored).

[0111] If the system is based on shallow anchor point recovery opportunity-triggered recovery, then B′=P_c+W_f (That is, based on the current second-level matching length, pull forward another fixed window) W_f (Data on size). Therefore, Formula 4 ensures that the recovery operation will not overdraw system resources indefinitely, and can dynamically match the most reasonable retrieval scale according to different triggering reasons. No matter which recoverable scenario, the length of the recovery will not exceed the total length of the processing request.

[0112] In step S206, a restricted prefetch request is initiated to the tertiary storage based on the second prefix matching status and the upper limit of the recovery range.

[0113] After determining the recoverable starting point and length of the missing prefix cache data, the system begins prefetching. In some embodiments, initiating a restricted prefetch request to the tertiary storage based on the second prefix matching state and the upper limit of the recovery range includes: first, determining the end node corresponding to the matching length in the second prefix matching state as the prefix matching anchor point; then, based on the upper limit of the recovery range, initiating a restricted prefetch request to the tertiary storage starting from this prefix matching anchor point.

[0114] In some embodiments, regardless of whether the recovery condition is triggered by prefix cache data degradation or a shallow anchor recovery opportunity, once the prefix matching anchor and the corresponding upper limit of the recovery range are determined, the subsequent processing flow for requesting data from the underlying tertiary storage can remain highly consistent at the underlying execution level. That is, they can use the broken prefix matching anchor as the addressing starting point to request subsequently lost prefix cache data from the tertiary storage.

[0115] In other embodiments, the system may also configure different priorities or data chunking fetch strategies for restricted prefetch requests based on different triggering conditions, such as assigning higher I / O priority to requests that have clearly degraded in order to accelerate their recovery.

[0116] In step S208, the missing prefix cache data obtained through the restricted prefetch request is written to the secondary storage, wherein the data length of the missing prefix cache data is limited by the upper limit of the recovery range.

[0117] Upon receiving a restricted prefetch request, the Level 3 storage can send back the corresponding historical prefix cache data stored internally. The system then writes this data into the Level 2 storage, changing its state from the underlying storage state to a recoverable state. Based on the recovery range upper limit determined in step S204, the actual length of data written to the Level 2 storage is controlled within the recovery range upper limit.

[0118] Therefore, through method 200, requests can be intercepted and processed under access control, and limited prefetch recovery can be performed. This ensures that the recovery benefits directly affect the currently processed request, avoiding the false recovery problem where subsequent requests indirectly benefit from the recalculation of the current request. At the same time, the recovery range can be limited by historical recoverable length or a fixed window, preventing unbounded prefetching from further increasing the pressure on secondary storage. Thus, the system can effectively control the impact on secondary storage while improving the success rate of prefix cache recovery.

[0119] In step S110, based on the restored prefix cache data in the secondary storage, the processing request is re-prefix matched, so that according to the third prefix matching status, the prefix cache data that is re-matched with the processing request in the secondary storage is loaded into the primary storage to admit and execute the processing request.

[0120] Specifically, after the restricted prefetching operation in step S108 is completed, the data pulled from the third-level storage has fallen into the second-level storage. At this point, the scheduler will query the prefix tree again for the same processing request in the next round of scheduling to rematch. The result of this matching is the third prefix matching state. This state reflects the latest cache hit situation that the processing request can actually utilize after the system's proactive rescue.

[0121] In some embodiments, admitting a processing request includes: transferring the processing request from a waiting queue to an execution state; and allocating computing resources located in primary storage to the processing request so that the processing request performs pre-filling processing.

[0122] As mentioned above, the admission mechanism is essentially the control flow for the system to allocate computing resources and memory. Transferring a processing request from the waiting queue (or resuming the suspended queue) to the execution state signifies that the request has officially gained scheduling permission from the system. Allocating computing resources includes allocating GPU computing core resources, batch processing slots, and the required primary storage (e.g., video memory) space for the request. The pre-filling process mentioned in this disclosure refers to the process in large language model inference where, for new lexical units in the prompt word sequence of the processing request input that do not hit the cache, the corresponding attention tensor data is calculated through forward propagation of the model. These intensive matrix operations only actually occur after resource allocation and formal admission are completed.

[0123] In some embodiments, loading prefix cache data that is re-matched with the processing request in the secondary storage to the primary storage to admit and execute the processing request, based on a third prefix matching state, includes: loading the matched prefix cache data in the secondary storage to the primary storage in response to a third prefix matching state indicating that the matching length of the prefix cache data that matches the processing request in the secondary storage is greater than zero; and admitting the processing request based on the third prefix matching state to perform pre-filling processing on the remaining unmatched portion of the processing request.

[0124] The above embodiment corresponds to the scenario where the system successfully recovers data from the third-level storage. If the third prefix matching state indicates that the matching length in the second-level storage has recovered to non-zero (or significantly longer than the second state), it means that the missing data has been successfully retrieved to the second-level storage. At this time, the system will utilize the existing load-back mechanism to physically move and load the matched prefix cache data from the second-level storage to the first-level storage via the bus, changing it from a recoverable state to a ready state that can be used directly. Subsequently, the system admits the request according to this deeper prefix matching state (i.e., the third prefix matching state). At this time, the request only needs to be recalculated for the remaining unmatched tokens (i.e., the suffix part), that is, to perform pre-filling processing. Since most of the historical data has been retrieved, the computational load of this pre-filling processing is greatly reduced.

[0125] In some embodiments, in response to a third prefix matching state indicating that the matching length of the prefix cache data matching the processing request in the secondary storage is zero, the processing request is admitted according to a second prefix matching state to perform pre-filling processing on the remaining unmatched portion of the processing request.

[0126] The above embodiment corresponds to a situation where the system attempted a recovery operation but ultimately failed to retrieve the data. In some cases (e.g., the relevant data in the lower-level three storage has just been cleaned up, or prefetching failed due to I / O failure), even if prefetching is performed, the third prefix matching status still shows that the matching length in the second-level storage is zero. In this case, to avoid the processing request being suspended indefinitely in a deadlock, the system can directly roll back and forcibly admit the request according to the actual available status detected before recovery (the second prefix matching status), allowing it to perform a large-scale pre-filling recalculation, thereby ensuring the continuity and robustness of the business logic.

[0127] In some embodiments, in response to determining that the prefix cache recovery condition is not met, pre-filling processing is performed on the remaining unmatched portion of the processing request based on the second prefix matching status admission processing request.

[0128] The above embodiment describes a processing branch where the recovery mechanism is not triggered. That is, in the previous step S106, the system, through state comparison, finds that neither cache hit degradation has occurred, nor has a high-value shallow anchor long request opportunity appeared (e.g., this could be a complete cold start request, or the request's deep cache has remained intact in secondary or primary storage during the waiting period). In this case, the system directly allows the processing request to proceed normally according to the latest observed state at the admission time (i.e., the second prefix matching state), to pre-fill the remaining unmatched portion, thereby ensuring that requests that do not require recovery can pass directly to the computation unit with maximum efficiency.

[0129] Therefore, by dynamically matching states at multiple stages—including queuing, pre-admission, and post-recovery—and by establishing judgment conditions for various branch scenarios such as recovery triggering, success, failure, or no recovery required, it can be ensured that each processing request is admitted in the best cached state it can obtain. This achieves clear hierarchical semantic flow and proactive recovery from abnormal states at the system level. The cached data scheduling method provided by the embodiments of this disclosure not only avoids incorrect interpretation of cached states but also minimizes unnecessary redundant calculations, significantly improving the throughput of processing requests and long-tail stability in high-concurrency environments.

[0130] Figure 3 An exemplary state machine diagram of the pre-admission recovery state transition for a processing request is shown according to an embodiment of the present disclosure. The diagram details the exemplary key state transitions between the processing request entering the waiting queue and entering pre-filled computation.

[0131] Once a request is enqueued, the system first enters the "Enqueued" state and records its historical recoverable state (i.e., the first prefix matching state mentioned above). The state then transitions to the "Recorded" state. Next, the request waits in the queue for admission, remaining in the "Waiting" state. Just before the request is admitted and executed, the system triggers a pre-admission rematch (i.e., determines the second prefix matching state), causing the request to enter the "Rematching" state. In this state, the system determines the subsequent flow branch based on the state comparison result: if the system determines that recovery has not been triggered (i.e., the prefix cache recovery condition is not met), the request will directly flow to the "Admitted" state; conversely, if the system detects a degradation or shallow anchor opportunity (i.e., the prefix cache recovery condition is met), the request will enter the "Intercepted" state (i.e., admission control is triggered, suspending the admission processing request).

[0132] In the "intercepted" state, if the recovery request is not initiated due to internal system limitations, the processing request will still flow as a fallback branch to the "admitted" state. If the system successfully initiates a Level 3 storage recovery request, the state changes to "Recovery Request Received." Subsequently, the current processing request remains in the queue, entering the "Recovery Waiting" state. During this period, the system performs a Level 3 storage prefetch operation (i.e., initiates a restricted prefetch request to Level 3 storage and writes to Level 2 storage). After the prefetch is completed, the state transitions to "Recovery Completed." Subsequently, the system performs a post-recovery re-match for this request, causing it to enter the "Post-Recovery Re-matching" state (i.e., determining the third prefix match status).

[0133] After this rematch, if the system finds that no valid secondary storage hit has been recovered (i.e., the third prefix matching status indicates a match length of zero in secondary storage), the processing request will be returned and flow to the normal "admitted" state. Conversely, if a secondary storage hit is detected as recovered (i.e., the third prefix matching status indicates a match length greater than zero in secondary storage), the "load back" state is triggered, and the recovered prefix cache data is loaded into primary storage via the physical bus. The processing request then flows to the optimal "admit with deep prefix" state. Finally, regardless of whether the request is in the "admit with deep prefix" state or the normal "admitted" state, the scheduler will allocate resources to it, allowing it to enter pre-filling computation with a deep prefix or the current prefix, and the state changes to "pre-filling" (i.e., performing pre-filling processing). After the pre-filling processing is completed, the processing request will continue with the subsequent decoding process or be declared complete directly.

[0134] By introducing key intermediate states such as "intercepted", "waiting for recovery", and "rematching after recovery", this state machine ensures that once the recovery operation is successfully initiated, the request will not directly enter the pre-filling calculation in the same round according to the degenerate shallow prefix, thereby completely avoiding the phenomenon of false recovery caused by a large number of unnecessary recalculations of the current request.

[0135] Figure 4 An interactive sequence diagram of pre-admission recovery of prefix cache data according to an embodiment of the present disclosure is shown. This sequence diagram illustrates in detail the signaling and data interaction processes between the processing requests in the multi-tiered storage architecture and the various logical scheduling modules of the system.

[0136] like Figure 4 As shown, the system may include logical components such as an admission scheduler, a prefix matching module, a degradation detection module, an admission control module, and a model executor. Requests are first entered into a waiting queue. During the enqueue phase, the admission scheduler calls the prefix matching module to perform initial or waiting-period prefix matching. The prefix matching module then returns the first prefix matching status (i.e., returns the historical matching length). P_o, H_o, R_oUpon receiving this state, the admission scheduler performs a self-call to record historical recoverable states.

[0137] When the request queuing process is finished and admission is about to begin, the admission scheduler calls the prefix matching module again to perform a pre-admission rematch. The prefix matching module then returns to the second prefix matching state (i.e., returns the current matching length). P_c, H_c, R_ c Subsequently, the admission scheduler submits the recorded historical matching status along with the current matching status to the degradation detection module for comparison. The degradation detection module performs a degradation or shallow anchor point determination and returns the determination result to the admission control module.

[0138] At this point, the system interaction flow enters different processing branches based on the judgment result. In the branch where pre-admission recovery is determined, the admission control module first sends a signaling to the admission scheduler to suspend the current request admission, thereby intercepting the request at the front end. Subsequently, the admission control module initiates a restricted recovery request to the tertiary storage based on the detected shallow anchor point. After receiving the request, the tertiary storage performs a prefetch operation, transferring the missing prefix cache data (prefetch key-value pairs) to the secondary storage, and sends a recovery completion notification to the admission control module after the operation is completed.

[0139] Upon receiving the recovery completion notification, the access control module triggers the prefix matching module to match the same processing request again, and receives the recovered secondary storage hit length (i.e., determines the third prefix matching status) returned by the prefix matching module. Based on this hit status, the access control module requests the secondary storage to load back, prompting the secondary storage to load the matched key-value pairs into the primary storage. Finally, the access control module notifies the model executor, causing the processing request to enter the pre-filling computation stage with a deep prefix.

[0140] Conversely, in branches where recovery is not required—that is, if the degradation detection module's judgment indicates that the prefix cache recovery condition is not met—the aforementioned restricted recovery and reload interaction is unnecessary. The access control module directly allows passage and notifies the model executor to enter pre-filling based on the current matching result.

[0141] Thus, through the close interaction between the above-mentioned logical modules and multi-layer storage, the system realizes a complete scheduling closed loop from state monitoring and admission interception to restricted prefetching and ready loading, thereby ensuring that processing requests can be executed in the optimal cache state.

[0142] To further verify the actual technical effect of the disclosed technical solution, experimental tests were conducted in a multi-level storage inference service environment (e.g., based on inference frameworks such as SGLang) that includes primary storage (e.g., GPU memory), secondary storage (e.g., host memory), and tertiary storage (e.g., Mooncake storage cache system).

[0143] In the test scenario, the model adopted a long-context, large-language model with a request concurrency of 64, and multiple processing requests shared or had similar prefixes. In actual operation, the system effectively handled two typical cache state anomaly scenarios. In an example where prefix cache data degradation occurred, the processing request recorded a deep recoverable prefix in the second-level storage upon enqueuing; however, during the queuing period, the second-level storage was evicted, causing its rematching result before admission to degrade to a shallow anchor with only a few hundred tokens, and the second-level storage hit length dropped to zero. Using the method disclosed herein, the system successfully detected this degradation and paused admission, initiating a limited recovery to the third-level storage starting from this shallow anchor. After prefetching, the hit length of the request was successfully restored to tens of thousands of tokens, ultimately entering pre-filling processing with a deep prefix, avoiding most of the redundant calculations for long prompt words. In another example where there was a shallow anchor recovery opportunity, although the processing request did not record a valid historical second-level storage hit, it exhibited characteristics of zero second-level storage hits, non-zero shallow anchors, and extremely long remaining prompt words before admission. The system also precisely triggered the restricted Level 3 storage recovery, and by requiring that the Level 1 storage hit must be greater than zero, it effectively avoided blindly initiating invalid underlying recovery for a complete cold start request.

[0144] Experimental data demonstrates the performance improvements brought about by this disclosed mechanism. After optimizing the scheduling and loadback strategies described above, the overall request throughput in the test environment increased from 0.3076 rps to 0.4530 rps, an improvement of approximately 47.3%; the output token throughput increased from 120.00 tok / s to 184.05 tok / s, an improvement of approximately 53.4%; and the overall cache hit rate of the system increased significantly from 0.4440 to 0.7993, an improvement of approximately 80.0%.

[0145] Regarding latency metrics, the average request latency decreased from 196.18 seconds to 134.12 seconds, a reduction of approximately 31.6%; the average Time To First Token (TTFT) latency decreased from 114.52 seconds to 79.88 seconds, a reduction of approximately 30.2%. Crucially, for long-tail latency metrics that significantly impact user experience, the TTFT latency for p90, p95, and p99 was significantly reduced by 46.4%, 45.0%, and 43.1%, respectively.

[0146] Those skilled in the art will understand that, because this disclosure effectively eliminates the phased deterioration phenomenon in the prior art where front-end request processing is extremely fast but back-end requests become extremely congested due to cache corruption, the overall request latency distribution becomes smoother and more stable. Therefore, there may be slight fluctuations or increases in local indicators such as the median, for example, changing from "fast front-end, crashing back-end" to "smoother and more stable overall." However, this reflects the improvement in the overall stability of the system. Considering indicators such as average latency, overall throughput, cache hit rate, and long-tail latency, this disclosure has demonstrated through experiments that it can effectively solve the cache hit degradation problem in high-concurrency long prompt word scenarios, significantly improving the overall stability of the model inference service and the utilization efficiency of computing resources.

[0147] In practical industrial applications, the dynamic cache data scheduling mechanism based on a multi-layer storage architecture provided in this disclosure has a wide range of application scenarios.

[0148] At the application level, the technical solution disclosed herein is widely applicable to large language model inference platforms employing multi-layered storage architectures, particularly suited to complex business scenarios characterized by high concurrency, long contexts, high request prefix similarity, and multi-level cache sinking and recovery mechanisms. Those skilled in the art should understand that this method does not rely on specific neural network model structures or specific programming language implementations at the underlying logic level, thus possessing universality and scalability. As a general scheduling optimization mechanism, it can be integrated into the admission scheduler, cache manager, or storage controller of various large model inference systems. In specific business implementations, this disclosure is particularly suitable for cutting-edge artificial intelligence scenarios with extremely high requirements for long context processing capabilities, such as code generation model services, long document intelligent question answering systems, retrieval augmentation generation (RAG) systems, and complex trajectory replay of intelligent agents.

[0149] From an economic and system value perspective, this disclosure significantly reduces repetitive matrix calculations during the pre-filling phase of long-prompt-word requests by intercepting and recovering lost prefix cache data, thereby greatly improving the effective utilization of core computing units (such as high-value computing resources like GPUs). This means that with the same hardware configuration, the system can process a larger scale of concurrent requests in parallel and significantly shorten the average user wait time. Therefore, this disclosure not only comprehensively improves the service stability and smoothness of long-context-related services but also effectively alleviates the high hardware cost pressure caused by blindly expanding computing nodes, maximizing the return on investment (ROI) of computing resources.

[0150] Figure 5 An exemplary block diagram of a data scheduling apparatus according to embodiments of the present disclosure is shown. It can be utilized... Figure 5 The device shown is used to perform Figure 1 Method 100 is described.

[0151] like Figure 5 As shown, in some embodiments, the data scheduling device 500 includes a first matching module 502, a second matching module 504, a recoverable detection module 506, a recovery module 508, and an execution module 510.

[0152] The first matching module 502 can be used to enter the waiting queue in response to a processing request and determine the first prefix matching status of the processing request.

[0153] The second matching module 504 can be used to enter the admission state in response to a processing request and determine the second prefix matching state of the processing request.

[0154] The recoverable detection module 506 can be used to determine whether the prefix cache recovery conditions are met based on the first prefix matching status and the second prefix matching status.

[0155] Recovery module 508 can be used to pause the admission processing request and initiate a recovery request to the tertiary storage in response to determining that the prefix cache recovery conditions are met, so as to restore the missing prefix cache data from the tertiary storage to the tertiary storage.

[0156] The execution module 510 can be used to re-match the prefix of the processing request based on the restored prefix cache data in the secondary storage, so as to load the prefix cache data that is re-matched with the processing request in the secondary storage into the primary storage according to the third prefix matching status, so as to admit and execute the processing request.

[0157] It should be understood that Figure 5 The various modules or units of the apparatus 500 shown can be connected to the reference. Figure 1 and Figure 2 The steps in methods 100 and 200 described correspond to each other. Therefore, the operations, features, and advantages described above for these methods also apply to apparatus 500 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0158] Those skilled in the art should understand that Figure 4 and Figure 5The modules shown (such as the admission scheduler, prefix matching module, degradation detection module, admission control module, and model executor) are merely exemplary logical divisions illustrating the interaction flow of embodiments of this disclosure and are not intended to strictly limit the specific implementation methods and protection scope of this disclosure. In practical applications, depending on the specific system architecture, programming framework, or hardware limitations, the functions of the above modules can be integrated into the same physical device or software process for centralized implementation, or they can be further divided into more sub-modules to be collaboratively completed by multiple distributed computing nodes. Furthermore, the names of the above modules are merely for clearly describing the logical functions of each signaling interaction link and should not be considered as a limitation on the names of technical features. Any software program, hardware circuit (such as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.), or hardware-software combination entity capable of implementing the same or equivalent interaction logic should be included within the protection scope of this disclosure.

[0159] According to one aspect of this disclosure, a processor is also provided. Figure 6 An exemplary block diagram of a processor according to an embodiment of the present disclosure is shown. Figure 6 The processor 600 shown may include a first memory 602, a second memory 604, a third memory 606, and an execution unit 608.

[0160] The first memory 602 can be used to store ready prefix cache data that can be directly involved in model calculation.

[0161] The second memory 604 is used to store recoverable prefix cache data that can be transferred to the first memory 602 by loading.

[0162] The third memory 606 is used to store the underlying prefix cache data fetched to the second memory 604 via restricted prefetch requests.

[0163] The execution unit 608 is coupled to the first memory 602, the second memory 604 and the third memory 606, and the execution unit 608 is configured to execute the cached data scheduling method according to the embodiments of the present disclosure in response to receiving a processing request.

[0164] It is understood that the processor 600 can be implemented to perform the data scheduling method according to embodiments of the present disclosure. For details regarding the specific implementation of the data scheduling method, please refer to the description above, which will not be repeated here.

[0165] According to one aspect of this disclosure, an electronic device is also provided. Figure 7 An exemplary block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 7As shown, the electronic device 1000 includes a memory 1020, a processor 1040, and a computer program stored in the memory. The memory 1020 and the processor 1040 may be coupled via an internal bus or other communication mechanism. The processor 1040 is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0166] Figure 8 A block diagram of an example computer device 2000 according to an embodiment of the present disclosure is shown.

[0167] The computer device 2000 may specifically correspond to the electronic device or processing system in the foregoing embodiments. It should be noted that... Figure 8 The components of the computer device 2000 shown are merely exemplary and not restrictive. Depending on the actual application requirements, the computer device 2000 may have other components or omit some components.

[0168] like Figure 8 As shown, computer device 2000 includes processing device 2010. Processing device 2010 can perform various appropriate actions and processes according to computer-executable instructions stored in read-only memory (ROM) 2020 or loaded from storage device 2080 into random access memory (RAM) 2030. For example, processing device 2010 can be configured to execute the above-referenced... Figures 1 to 4 The described method for scheduling cached data. Various programs and data required for the operation of the computer device 2000 can also be temporarily stored in RAM 2030. The processing unit 2010, ROM 2020, and RAM 2030 are interconnected and exchange data via bus 2040. Input / output (I / O) interface 2050 is also connected to bus 2040.

[0169] In some embodiments, the processing device 2010 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In implementing embodiments of this disclosure, the processing device 2010 may have a built-in or coupled data scheduling device as described above for scheduling cached data.

[0170] Typically, the following devices can be connected to the I / O interface 2050: input devices 2060 including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 2070 including, for example, liquid crystal displays (LCDs), light-emitting diode displays (LEDs), speakers, vibrators, etc.; storage devices 2080 including, for example, magnetic tapes, hard disks, solid-state drives (SSDs), flash memory cards, etc.; and communication devices 2090.

[0171] The storage device 2080 can serve as a high-capacity, non-volatile storage medium for persistently storing the operating system, applications, and data such as large-scale neural network weights, model weights, requested data, and prefix cache data. The communication device 2090 allows the computer device 2000 to exchange data with other devices (such as cloud servers, other computing nodes in a distributed computing cluster, etc.) via a network, either wirelessly or via wired communication. The communication device 2090 may include a local area network (LAN) card, a wireless local area network (WLAN) card, a Bluetooth module, a near field communication (NFC) module, a cellular communication module, etc.

[0172] Although Figure 8 A computer device 2000 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the computer device 2000 may alternatively implement or possess more or fewer devices. For example, when the computer device 2000 is used as an AI computing accelerator card or server node in a data center, it may not include input devices 2060 and output devices 2070 for human-computer interaction such as touch screens and keyboards, but may mainly consist of a processing unit 2010, high-bandwidth memory, and high-speed communication interfaces.

[0173] The functions of the systems, apparatuses, modules, or units described above herein can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on. Furthermore, the methods described in this disclosure can also be implemented by software or microcode containing computer-executable instructions that, when executed by the processing device 2010, cause the machine to perform the operations defined in this disclosure.

[0174] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided. Figure 9 An exemplary block diagram of a computer-readable storage medium according to embodiments of the present disclosure is shown. Figure 9As shown, a computer-readable storage medium 3000 stores computer-executable instructions 3010, which, when executed by a processor, implement the steps of any of the method embodiments described above.

[0175] In some embodiments, the computer-readable storage medium 3000 may include any combination of one or more computer program products. For example, without limitation, the non-transitory computer-readable storage medium may include volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0176] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0177] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited by these embodiments or examples, but only by the claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A cached data scheduling method, applied to a storage architecture including primary storage, secondary storage, and tertiary storage, characterized in that, The method includes: In response to a processing request entering a waiting queue, the first prefix matching status of the processing request is determined; In response to the processing request entering the admission pending state, the second prefix matching state of the processing request is determined; Based on the first prefix matching status and the second prefix matching status, determine whether the prefix cache recovery conditions are met; In response to determining that the prefix cache recovery conditions are met, a recovery request is initiated to the third-level storage to restore the missing prefix cache data from the third-level storage to the second-level storage; Based on the restored prefix cache data in the secondary storage, the processing request is re-prefix matched, and according to the third prefix matching status, the prefix cache data that is re-matched with the processing request in the secondary storage is loaded into the primary storage to admit and execute the processing request.

2. The cached data scheduling method according to claim 1, characterized in that, The first prefix matching state includes: a first-level matching length and a first-level matching length, wherein the first-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the waiting queue, and the first-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the waiting queue. The second prefix matching state includes: a second first-level matching length and a second second-level matching length, wherein the second first-level matching length is the matching length between the processing request and the prefix cache data in the first-level storage when the processing request enters the admission state, and the second second-level matching length is the matching length between the processing request and the prefix cache data in the second-level storage when the processing request enters the admission state.

3. The cached data scheduling method according to claim 2, characterized in that, The first prefix matching state also includes the sum of the first level matching length and the first level matching length, and the second prefix matching state also includes the sum of the second level matching length and the second level matching length.

4. The cached data scheduling method according to claim 3, characterized in that, The step of determining whether the prefix cache recovery condition is met based on the first prefix matching state and the second prefix matching state includes: determining that the prefix cache recovery condition is met when it is determined that the prefix cache data has degraded or that there is a shallow anchor point recovery opportunity.

5. The cached data scheduling method according to claim 4, characterized in that, Prefix cache data is considered to have degraded when the following conditions are met: The sum of the first-level matching length and the first-level matching length is greater than the sum of the second-level matching length and the second-level matching length.

6. The cached data scheduling method according to claim 5, characterized in that, The conditions for determining that the prefix cache data has degraded also include: The length of the first secondary match is greater than zero; and The second-level matching length is zero.

7. The cached data scheduling method according to claim 4, characterized in that, When it is determined that the prefix cache data has not degraded, a judgment is made on whether there is a shallow anchor point recovery opportunity.

8. The cached data scheduling method according to claim 4, characterized in that, The existence of a shallow anchor point recovery opportunity is determined when the following conditions are met: The second-level matching length is less than or equal to a preset shallow prefix threshold; and The remaining unmatched length of the processing request is greater than or equal to a preset recovery trigger threshold, wherein the remaining unmatched length of the processing request is the total length of the processing request minus the sum of the second-level matching length and the second-level matching length.

9. The cached data scheduling method according to claim 8, characterized in that, The conditions for determining the existence of the shallow anchor point restoration opportunity also include: The second and third level matching lengths are zero; and The second-level matching length is greater than zero.

10. The cached data scheduling method according to claim 4, characterized in that, The step of initiating a recovery request to the tertiary storage to restore the missing prefix cache data from the tertiary storage to the secondary storage includes: Suspend the access processing request; Determine the upper limit of the recovery range for the prefix cache data from the three-level storage; Based on the second prefix matching status and the upper limit of the recovery range, a restricted prefetch request is initiated to the third-level storage; The missing prefix cache data obtained through the restricted prefetch request is written to the secondary storage, wherein the data length of the missing prefix cache data is limited by the upper limit of the recovery range.

11. The cached data scheduling method according to claim 10, characterized in that, The step of initiating a restricted prefetch request to the third-level storage based on the second prefix matching status and the upper limit of the recovery range includes: The terminal node corresponding to the matching length in the second prefix matching state is determined as the prefix matching anchor point; Based on the upper limit of the recovery range, a restricted prefetch request is initiated to the third-level storage, starting from the prefix matching anchor point.

12. The cached data scheduling method according to claim 10, characterized in that, The determination of the upper limit of the recovery range from the prefix cache data in the three-level storage includes: In response to determining that the prefix cache data has degraded, the upper limit of the recovery range is determined to be the smaller value between the total length of the processing request and the sum of the first-level matching length and the first-level matching length; and In response to determining that there is a shallow anchor point recovery opportunity, the upper limit of the recovery range is determined to be the smaller value between the total length of the processing request and the sum of the second-level matching length and the preset recovery window length.

13. The cached data scheduling method according to claim 1, characterized in that, The step of loading the prefix cache data that is re-matched with the processing request in the secondary storage into the primary storage according to the third prefix matching status, so as to admit and execute the processing request, includes: In response to the third prefix matching status indicating that the matching length of the prefix cache data matching the processing request in the secondary storage is greater than zero, the matching prefix cache data in the secondary storage is loaded into the primary storage; and The processing request is admitted based on the third prefix matching status to perform pre-filling processing on the remaining unmatched portion of the processing request.

14. The cached data scheduling method according to claim 1, characterized in that, The method further includes: In response to the third prefix matching state indicating that the matching length of the prefix cache data matching the processing request in the secondary storage is zero, the processing request is admitted according to the second prefix matching state to perform pre-filling processing on the remaining unmatched portion of the processing request.

15. The cached data scheduling method according to claim 1, characterized in that, The method further includes: In response to determining that the prefix cache recovery condition is not met, the processing request is admitted according to the second prefix matching status to perform pre-filling processing on the remaining unmatched portion of the processing request.

16. The cached data scheduling method according to claim 1, characterized in that, The access processing request includes: Transfer the processing request from the waiting queue to the execution state; and Allocate computing resources located in the primary storage to the processing request so that the processing request can perform pre-population processing.

17. The cached data scheduling method according to any one of claims 1-16, characterized in that, The processing request includes an inference request for a large language model, and the prefix cache data includes attention key-value pair cache data, wherein the attention key-value pair cache data is attention tensor data calculated for the prompt words of the inference request during the pre-filling process of the large language model.

18. A data scheduling device, comprising: The first matching module is used to determine the first prefix matching status of the processing request in response to entering the waiting queue. The second matching module is used to determine the second prefix matching state of the processing request in response to the processing request entering the admission state. The recoverable detection module is used to determine whether the prefix cache recovery conditions are met based on the first prefix matching state and the second prefix matching state. The recovery module is used to pause the admission processing request in response to determining that the prefix cache recovery conditions are met, and to initiate a recovery request to the tertiary storage to restore the missing prefix cache data from the tertiary storage to the secondary storage; The execution module is used to re-match the prefix of the processing request based on the restored prefix cache data in the secondary storage, and load the prefix cache data that is re-matched with the processing request in the secondary storage into the primary storage according to the third prefix matching status, so as to admit and execute the processing request.

19. A processor, comprising: The first memory is used to store ready prefix cache data that can be directly used in model calculations; The second memory is used to store recoverable prefix cache data that can be transferred into the first memory by loading; The third memory is used to store the underlying prefix cache data fetched to the second memory through restricted prefetch requests; An execution unit, coupled to the first memory, the second memory, and the third memory, is configured to execute the cached data scheduling method according to any one of claims 1-17 in response to receiving a processing request.

20. An electronic device, comprising: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the cache data scheduling method according to any one of claims 1-17.

21. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the cached data scheduling method according to any one of claims 1-17.

Images