Key-value cache data processing method and electronic device

Abstract

The application provides a key-value cache data processing method and electronic equipment, which can be applied to the fields of model inference and computer technology. The method comprises the following steps: based on the access priority of a key-value cache data block of a to-be-inferred task, the key-value cache data block is allocated to an initial storage layer in a multi-level storage layer for storage; according to a hotness evaluation result of the key-value cache data block and a positional relationship between the initial storage layer and an intermediate storage layer in the multi-level storage layer, a software abstraction interface is used to select and call a hardware transmission link to migrate the key-value cache data block in the initial storage layer to the intermediate storage layer, wherein the hotness evaluation result indicates a migration direction of the key-value cache data block; in response to an access request for the key-value cache data block, the key-value cache data block in the intermediate storage layer is migrated to a target storage layer in the multi-level storage layer, so as to decode the key-value cache data block in the target storage layer and generate an inference result.

Description

Technical Field

[0001] This invention relates to the fields of model reasoning and computer technology, and more specifically to a method for processing key-value cached data and an electronic device. Background Technology

[0002] In large-scale model inference, the efficient management and transmission of key-value cache data determines inference latency, throughput, and resource utilization. Related technologies primarily employ pagination attention mechanisms to divide key-value cache data into fixed-size pages, enabling memory fragmentation reclamation and dynamic management. The decoding phase utilizes a continuous batch processing mechanism to dynamically adjust task batches to improve GPU utilization. A tensor parallel strategy is employed to distribute model layers across different GPUs, with key-value cache data stored in the local storage resources of each computing node.

[0003] However, the storage costs of related technologies are high and there is a lack of actual storage needs for different scenarios; data management is neglected from the perspective of the complete state of the data and the hardware level; there is a lack of automated synchronization mechanisms, the deployment is complex, and the data is prone to inconsistency issues. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method for processing key-value cached data and an electronic device.

[0005] According to a first aspect of the present invention, a method for processing key-value cache data is provided, comprising: allocating key-value cache data blocks to an initial storage layer in a multi-level storage layer for storage based on the access priority of key-value cache data blocks of a task to be inferred, wherein the access priority is determined by the access frequency of the key-value cache data blocks and the cost information of storing the key-value cache data blocks; migrating key-value cache data blocks in the initial storage layer to the intermediate storage layer using a software abstraction interface based on the heat assessment result of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, wherein the heat assessment result indicates the migration direction of the key-value cache data blocks; and migrating key-value cache data blocks in the intermediate storage layer to a target storage layer in the multi-level storage layer in response to an access request for the key-value cache data blocks, so as to perform decoding based on the key-value cache data blocks in the target storage layer and generate an inference result.

[0006] A second aspect of the present invention provides a processing apparatus for key-value cache data in model inference, comprising: a storage module, configured to allocate key-value cache data blocks to an initial storage layer in a multi-level storage layer for storage based on the access priority of the key-value cache data blocks of the task to be inferred, wherein the access priority is determined by the access frequency of the key-value cache data blocks and the cost information of storing the key-value cache data blocks; a selection module, configured to select and invoke a hardware transmission link to migrate key-value cache data blocks in the initial storage layer to the intermediate storage layer using a software abstraction interface, based on the heat evaluation result of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, wherein the heat evaluation result indicates the migration direction of the key-value cache data blocks; and a migration module, configured to migrate key-value cache data blocks in the intermediate storage layer to a target storage layer in the multi-level storage layer in response to an access request for the key-value cache data blocks, so as to perform decoding based on the key-value cache data blocks in the target storage layer and generate inference results.

[0007] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0008] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0009] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0010] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0011] Figure 1 The diagram illustrates a method for processing key-value cached data according to an embodiment of the present invention, and an application scenario of an electronic device.

[0012] Figure 2 A flowchart illustrating a method for processing key-value cache data according to an embodiment of the present invention is shown;

[0013] Figure 3 A schematic diagram illustrating an example of the migration process of key-value cache data blocks between different storage layers according to an embodiment of the present invention is shown.

[0014] Figure 4A structural block diagram of a device for processing key-value cache data in model inference according to an embodiment of the present invention is shown;

[0015] Figure 5 A block diagram of an electronic device suitable for implementing a key-value cache data processing method according to an embodiment of the present invention is shown. Detailed Implementation

[0016] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0018] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0019] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0020] The following are explanations or contents of the technical terms related to this invention.

[0021] Key-Value Cache (KV-cache) is used to store the "key" and "value" data generated by the attention mechanism during large model inference. Remote Direct Memory Access (RDMA), a hardware-level network technology, allows direct access to remote node memory without CPU intervention. A Worker is a node unit in the distributed inference cluster; each Worker can contain multiple GPUs and is responsible for executing part of the inference tasks and data storage and transmission. Metadata provides auxiliary information describing KV-cache data, including data identifier, storage location, data size, and lifecycle status, supporting KV-cache management and scheduling. The Key-Value Cache Block Manager (KvBlockManager) is responsible for the overall management of key-value cache blocks across the four storage layers, implementing lifecycle management and data migration scheduling for data blocks.

[0022] The inference transport library, acting as a software abstraction layer, encapsulates the underlying hardware transport technology, providing a unified interface for the cross-device or cross-node transport of key-value cache data blocks. The event plane, implemented using message queues, automatically synchronizes the metadata and lifecycle events of key-value cache data blocks, ensuring data consistency across nodes. Prefill-Decode (PD) separation separates the prefill and decoding phases of large model inference, avoiding resource contention between phases and improving inference efficiency.

[0023] In some examples, after implementing separate deployment of prefill and decode, prefill instances are decoupled from decode instances, and the computed key-value cache is reused locally through a prefix hash matching mechanism to reduce redundant computation.

[0024] For example, by supporting multiple parallel strategies, including tensor parallelism and expert parallelism, full-switched communication within expert parallel groups is achieved through remote direct memory access during hybrid expert model inference. Key-value cache data is stored in the local graphics processing unit (GPU) memory and host memory of each node. Multiple GPUs within the same node transmit data via a high-speed serial point-to-point data transmission channel (Peripheral Component Interconnect Express, PCIe), while cross-node data transmission relies on remote direct memory access. Metadata is stored in a centralized in-memory database (Remote Dictionary Server, Redis), and each worker node needs to periodically query Redis to update metadata information.

[0025] However, the storage layer in the relevant technology has limitations. In scenarios with long contexts (such as 128k tokens), key-value cache data needs to be frequently swapped in and out of the relatively limited storage layer, or key-value cache data is frequently discarded due to insufficient video memory, resulting in increased redundant calculations and inference delays.

[0026] Cross-device or cross-node transmission is inefficient. Data transfer between multiple GPUs within the same node relies on the PCIe bus, resulting in significant bandwidth bottlenecks. Data transfer between GPUs on different nodes requires CPU relay, leading to high latency and high CPU resource consumption. Furthermore, metadata synchronization relies on centralized storage. Sharing key-value cache data across nodes requires configuring communication protocols and synchronizing metadata, resulting in high deployment complexity and a high risk of data inconsistency issues.

[0027] In related technologies, even if cross-node key-value cache data transfer supports remote memory access technology, multiple GPUs within the same node still rely on PCIe and lack high-speed interconnect technology, thus limiting transmission efficiency. In traditional key-value cache data management systems, storage-level scheduling and data transmission technology are disconnected, making it difficult to dynamically select the optimal transmission path based on data importance, resulting in low resource utilization.

[0028] In view of this, embodiments of the present invention provide a method for processing key-value cache data and an electronic device. The method includes: allocating key-value cache data blocks to an initial storage layer in a multi-level storage layer for storage based on the access priority of the key-value cache data blocks of the task to be inferred, wherein the access priority is determined by the access frequency of the key-value cache data blocks and the cost information of storing the key-value cache data blocks; according to the heat evaluation result of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, using a software abstraction interface, selecting and calling a hardware transmission link to migrate the key-value cache data blocks in the initial storage layer to the intermediate storage layer, wherein the heat evaluation result indicates the migration direction of the key-value cache data blocks; and in response to an access request for the key-value cache data blocks, migrating the key-value cache data blocks in the intermediate storage layer to a target storage layer in the multi-level storage layer, so as to perform decoding based on the key-value cache data blocks in the target storage layer and generate an inference result.

[0029] According to embodiments of the present invention, key-value cache data blocks are allocated to the initial storage layer based on access priority. Based on heat assessment results and location relationships, a software abstraction interface drives data migration as needed between the initial storage layer, intermediate storage layer, and target storage layer. This expands the effective storage layers, improves cache hit rate in long-context scenarios, reduces data loss and redundant computation due to insufficient GPU memory, and thus reduces inference latency. The software abstraction interface automatically selects and invokes the optimal hardware transmission link based on the location relationships between storage layers, enabling high-speed direct connections within the same node and remote direct memory access across nodes. This avoids traditional PCIe bandwidth bottlenecks and CPU relay overhead, improving data transmission efficiency across devices and nodes.

[0030] It is understandable that by deeply coordinating data heat assessment, migration scheduling and underlying hardware transmission path selection, the scheduling decision of storage location can dynamically adapt to the optimal data transmission path, realizing the joint optimization of storage resources and network bandwidth, improving the overall resource utilization, and simplifying the complexity of system deployment and data consistency maintenance through the on-demand migration mechanism in response to access requests.

[0031] Figure 1 The diagram illustrates a method for processing key-value cached data and an application scenario of an electronic device according to an embodiment of the present invention.

[0032] like Figure 1 As shown, the application scenario according to this embodiment may include inference model 10 and computing nodes. The computing nodes may be physically independent server units, and there may be multiple computing nodes, for example, including computing node 1 and computing node 2.

[0033] Computing node 1 and computing node 2 each contain at least one multi-level storage layer. For example, computing node 1 contains multi-level storage layer 11 and multi-level storage layer 12, and computing node 2 contains multi-level storage layer 21.

[0034] Each of the multi-level storage layer 11, multi-level storage layer 12 and multi-level storage layer 21 can contain storage layers with different access priorities, such as storage layer G1, storage layer G2, storage layer G3 and storage layer G4.

[0035] The data block access priority of storage layers G1, G2, G3, and G4 decreases sequentially, while their storage capacity increases sequentially. Storage layer G1 can be the graphics processor's video memory, storage layer G2 can be the central processing unit's memory, storage layer G3 can be a local solid-state drive, and storage layer G4 can be a remote distributed object storage.

[0036] For example, after receiving the text to be inferred from the user input, the inference model 10 assigns the text to be inferred to computing node 1 for processing. Computing node 1 processes the text to be inferred, generates a corresponding key-value cache data block, calculates the access priority of the data block, and assigns the data block to storage layer G1 (graphics processor memory) in the multi-level storage layer 11 of computing node 1 for initial storage according to the priority matching rules.

[0037] It should be understood that Figure 1 The number of inference models, compute nodes, and multi-level storage shown is merely illustrative. Depending on implementation needs, any number of inference models, compute nodes, and multi-level storage can be used.

[0038] Figure 2 A flowchart of a method for processing key-value cached data according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, the key-value cache data processing method of this embodiment includes operations S210 to S230.

[0040] In operation S210, based on the access priority of the key-value cache data blocks of the task to be inferred, the key-value cache data blocks are allocated to the initial storage layer in the multi-level storage layer for storage. The access priority is determined by the access frequency of the key-value cache data blocks and the cost information for storing the key-value cache data blocks.

[0041] In embodiments of the present invention, the inference task can refer to a specific request unit submitted to the inference model for processing, including an input sequence (prompt), with the goal of generating a corresponding output sequence (response) to obtain the inference result. The key-value cache data block can be a set of key and value tensors calculated and cached by the model's attention mechanism for each token of the input sequence and the generated output sequence during the autoregressive decoding process of the large language model.

[0042] Access priority can be a numerical metric used to quantify the order and importance of key-value cache data blocks. Access frequency can be the number of times a key-value cache data block is referenced during the model decoding process. Cost information can be a quantified cost metric corresponding to the storage of a key-value cache data block in a specific storage layer. A multi-level storage layer can be a unified storage system constructed through software abstraction of various physical storage media with different performance, capacity, and cost. The initial storage layer can be the first physical storage location determined based on the initial access priority of the key-value cache data block after its generation.

[0043] For example, the access frequency and cost information of the current key-value cache data block are input into the priority calculation function to obtain the access priority value. Based on the preset matching rules in the system, the difference between the priority of the current data block and each storage layer is calculated, and the storage layer with the smallest difference is selected as the initial storage layer.

[0044] In operation S220, based on the heat assessment results of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, the software abstraction interface is used to select and invoke the hardware transmission link to migrate the key-value cache data blocks in the initial storage layer to the intermediate storage layer. The heat assessment results indicate the migration direction of the key-value cache data blocks.

[0045] In embodiments of the present invention, the heat assessment result can be a quantitative determination of the current access value of a key-value cache data block, for example, calculated based on dynamic indicators such as the recent access frequency, last access time, and associated task priority of the key-value cache data block. This result includes not only the heat score of the data block but also the expected migration direction. The location relationship can be a physical or logical topology connection state, which can be used to determine whether the initial storage layer and the intermediate storage layer are located within the same computing node. The software abstraction interface can be a programming interface layer for the application, used to abstract and encapsulate the differences between various underlying hardware transmission links. Hardware transmission links can be physical or logical data transmission channels connecting two storage layers; each link has specific performance (bandwidth, latency) and applicable scenarios. The intermediate storage layer can be the storage level where the target key-value cache data block resides before the access request.

[0046] For example, if a change in the popularity assessment result of a key-value cache data block is detected, the data block needs to be migrated from storage layer A to storage layer B according to the policy. A migration request is sent to the software abstraction interface, carrying the data block identifier, the initial storage layer identifier, and the intermediate storage layer identifier. After receiving the request, the software abstraction interface queries the storage layer resource topology table, resolves the positional relationship between the initial and intermediate storage layer identifiers, identifies candidate hardware transmission links, selects the optimal transmission link from the candidate hardware transmission links, and calls the corresponding underlying hardware driver library to pass in the key-value cache data block and its metadata.

[0047] In operation S230, in response to an access request for a key-value cache data block, the key-value cache data block in the intermediate storage layer is migrated to the target storage layer in the multi-level storage layer, so as to decode based on the key-value cache data block in the target storage layer and generate an inference result.

[0048] In embodiments of the present invention, an access request can be an explicit instruction or signal that requires reading or using a specific key-value cache data block to continue or complete a certain inference task. The target storage layer can be the final storage location to which the key-value cache data block is migrated in response to the current access request. This storage layer has high bandwidth and low latency characteristics and is the storage medium directly relied upon for performing the actual model decoding calculation. Decoding can be an autoregressive decoding process during model inference.

[0049] For example, upon receiving an access request with a data block identifier, the system queries the global metadata directory based on the required data block identifier to confirm that the data block currently resides in a certain intermediate storage layer. Then, based on the priority of the requesting computation task, the current system load, and the size of the target data block, the system initiates data block migration, requesting and reserving a contiguous space of matching size from the resource manager of the target storage layer (such as the video memory of a specified GPU). Subsequently, a migration command is issued through a unified software abstraction interface. The software abstraction interface automatically selects the optimal hardware transmission link based on the positional relationship between the intermediate and target storage layers, performing high-bandwidth data transmission to completely migrate the key-value cache data block from the intermediate storage layer to the reserved location in the target storage layer.

[0050] According to embodiments of the present invention, key-value cache data blocks are allocated to the initial storage layer based on access priority. Based on heat assessment results and location relationships, a software abstraction interface drives data migration on demand between the initial storage layer, intermediate storage layer, and target storage layer. This significantly expands the effective storage layers, improves cache hit rate in long-context scenarios, reduces data loss and redundant computation due to insufficient GPU memory, and thus reduces inference latency. The software abstraction interface automatically selects and invokes the optimal hardware transmission link based on the location relationships between storage layers, enabling high-speed direct connections within the same node and remote direct memory access across nodes. This avoids traditional PCIe bandwidth bottlenecks and CPU relay overhead, improving data transmission efficiency across devices and nodes. Deeply coordinating data heat assessment, migration scheduling, and underlying hardware transmission path selection allows storage location scheduling decisions to dynamically adapt to the optimal data transmission path, achieving joint optimization of storage resources and network bandwidth, improving overall resource utilization. Simultaneously, the on-demand migration mechanism in response to access requests simplifies system deployment and data consistency maintenance complexity.

[0051] According to an embodiment of the present invention, the multi-level storage layer includes: a first storage layer serving as graphics processor memory, used to store key-value cache data blocks with an access frequency greater than a first threshold; a second storage layer serving as central processing unit memory, used to store key-value cache data blocks with an access frequency less than or equal to the first threshold and greater than a second threshold; a third storage layer serving as a local solid-state drive, used to store key-value cache data blocks with an access frequency less than or equal to the second threshold and greater than a third threshold; and a fourth storage layer serving as remote object storage, used to store key-value cache data blocks with an access frequency less than or equal to the third threshold.

[0052] In embodiments of the present invention, the multi-level storage layer may include four levels of storage layers, from the first storage layer to the fourth storage layer, with the access performance of each storage layer decreasing sequentially but the storage capacity increasing sequentially. The threshold corresponding to the access frequency can be a preset critical value used to quantify and classify the access frequency of key-value cache data blocks, which can be specifically obtained according to actual needs or experiments.

[0053] For example, a four-level storage tier can be used to achieve hierarchical storage and dynamic migration of key-value cache data blocks. The four storage tiers can be designated G1, G2, G3, and G4. KV-cache data blocks can be allocated to storage tiers G1 through G4 based on data frequency, achieving a balance between efficient storage and cost.

[0054] G1 can be GPU memory used to store hot KV-cache data blocks (such as data frequently accessed in the last 10 minutes), supporting low latency and high bandwidth access to meet the real-time inference requirements of the decoding stage; G2 can be CPU memory used to store less hot KV-cache data blocks (such as data accessed in the last 30 minutes), serving as a transitional layer between G1 and G3 to balance access latency and storage costs; G3 can be local or pooled solid-state drives used to store cold data (such as data not accessed for 30 minutes to 2 hours), leveraging the large capacity of solid-state drives to reduce storage costs, and migrating to G1 or G2 via high-speed interconnect technology or Remote Direct Memory Access (RDMA) when needed; G4 can be remote storage used to store ultra-cold data (such as data not accessed for more than 2 hours), employing distributed object storage to further reduce long-term storage costs, and transferring to the local layer via RDMA during migration.

[0055] It is understandable that by associating access frequency with thresholds and mapping them to four levels of hardware with clear performance and cost gradients, a resource allocation strategy based on popularity is implemented. This ensures that GPU memory overhead is precisely used to serve the most frequent accesses, while massive amounts of low-frequency data are automatically diverted to low-cost media, thereby minimizing the total cost of ownership of storage at the system level.

[0056] According to an embodiment of the present invention, the method further includes: mapping the access priority to a target value within a preset range as a base heat value; determining the ratio between the used capacity of each storage layer and the total capacity in the multi-level storage layer as the capacity utilization rate of each storage layer; querying the type-coefficient mapping table according to the trigger signal type in the migration instruction to obtain the adjustment coefficient corresponding to the trigger signal type; and using preset weights to perform a weighted summation of the base heat value, the cost information of each storage layer, the capacity utilization rate, and the adjustment coefficient to obtain the heat evaluation result.

[0057] In embodiments of the present invention, elements influencing the data block heat assessment result may include data access priority, storage layer cost information, trigger signal type, and storage layer capacity utilization. The base heat value can be a benchmark score obtained by mapping access priority to a standardized numerical range (e.g., [0,1]). The capacity utilization rate can be a real-time indicator of the resource stress of each storage layer in a multi-level storage system, and its value is the ratio of the used capacity of that storage layer to the total capacity. The trigger signal type can define whether data migration is proactive or reactive, determining the optimization direction of the assessment. The preset weight can be a predefined parameter used to control the relative importance of the base heat value, cost information, capacity utilization rate, and adjustment coefficient in the final assessment result, indicating preferences for different optimization objectives (performance, cost, load balancing). Specific values ​​can be determined according to actual needs and are not limited here.

[0058] Table 1 shows the access priority, storage tier cost information, trigger signal type, and storage tier capacity utilization.

[0059] Table 1

[0060]

[0061] According to embodiments of the present invention, by incorporating multiple key dimensions such as access priority, cost information, capacity utilization, and trigger signal type into unified quantitative information, the one-sidedness of traditional single-indicator (such as least recently used) decision-making is overcome. Data migration decisions are based on a comprehensive assessment of the data's intrinsic value, storage economy, system real-time load, and specific operational objectives. The decision-making process is upgraded from experience-driven to data-driven, improving the rationality and global optimality of data block migration decisions.

[0062] For example, by incorporating capacity utilization as a dynamic factor into the heat assessment factor, an inherent negative feedback adjustment capability is achieved. When a high-performance storage layer is overloaded, its capacity utilization factor will reduce the score of the layer to be migrated, automatically guiding new data or migrating data to the layer with less pressure, effectively avoiding the overload of a single node and enhancing the overall load balancing capability and stability of the system.

[0063] According to an embodiment of the present invention, based on the access priority of the key-value cache data block of the task to be inferred, the key-value cache data block is allocated to the initial storage layer in a multi-level storage layer for storage, including: obtaining cost information of each storage layer in the multi-level storage layer, the cost information including unit capacity cost information and access latency information of the storage medium in each storage layer; determining the matching degree between the access priority and the cost information of each storage layer to obtain multiple matching degrees; selecting the storage layer corresponding to the peak matching degree from the multiple matching degrees as the initial storage layer, so as to store the key-value cache data block in the initial storage layer.

[0064] In embodiments of the present invention, unit capacity cost information may be, for example, the purchase or rental cost per GB of memory or video memory, and access latency information may be the average or tail latency information provided by the storage layer for data read and write. Matching degree may be used to measure the degree of agreement or fit between access priority and the cost information of a certain storage layer.

[0065] For example, when a key-value cache data block is generated or loaded for the first time, it can be preferentially allocated to the highest-level storage pool whose priority matches the hardware performance, as shown in the following formula (1):

[0066] (1);

[0067] Where P can be the priority score of the data to be stored. It can represent the hardware performance score of the storage layer (the lower the cost, the higher the performance score), that is, select the storage layer whose performance score is closest to the data priority, arg min is the minimum value operator of the independent variable, Pool∈{G1,G2,G3} can be a set of candidate storage pools, and Target Pool can be the target storage pool selected to store the data.

[0068] For example, when P=1 (the key-value cache data required for the current iteration layer): The value closest to P=1 is assigned to G1; when P=0.6 (potentially reused in the short term but not required in real time): The value matches P=0.6 perfectly, therefore it is assigned to G2; when P=0.2 (long-term archived data): It is a perfect match for P=0.2, therefore it is assigned to G3.

[0069] According to embodiments of the present invention, by making precise matching decisions based on forward-looking access priorities and objective cost information at the first moment of data block generation, the data is placed in the most cost-effective storage environment from the initial stage, fundamentally avoiding the frequent and expensive cross-level data migration caused by improper initial placement in traditional methods, and significantly reducing the scheduling overhead within the system.

[0070] According to an embodiment of the present invention, migrating a key-value cache data block in an initial storage layer to an intermediate storage layer includes: migrating the key-value cache data block stored in the initial storage layer to the intermediate storage layer in response to one of the following events: detecting that the capacity status of the initial storage layer indicates that the remaining capacity of the initial storage layer is less than a capacity threshold; receiving an access request for a target key-value cache data block in the initial storage layer.

[0071] In embodiments of the present invention, capacity status can be represented by capacity utilization. A capacity threshold can be used to determine the critical value for assessing the resource scarcity of the storage layer; the specific value can be determined based on actual needs.

[0072] For example, when the current high-priority storage layer G1 has insufficient capacity (M=1), or when a trigger signal indicating "data no longer needs to reside" is received (T... offload In this case, the data block is triggered to migrate to a lower priority storage layer (e.g., G2), and the migration path is shown in the following formula (2):

[0073] (2);

[0074] Among them, G2 and G3 are storage layers with lower priority than G1. G1This refers to the capacity utilization of storage layer G1. When the model needs to access data blocks in a lower-priority storage layer (such as key-value cache data blocks in G2 / G3), a "data needs to be reused" trigger signal (T) is issued. onboard This triggers the migration of data to a higher-priority storage layer (e.g., G1), and the migration path is shown in formula (3) below:

[0075] (3);

[0076] According to embodiments of the present invention, by monitoring capacity status and determining capacity thresholds, data cleanup can be proactively and proactively initiated before high-performance storage layer space resources become scarce. This avoids passive, hasty, and even data-loss-inducing overflow processing, ensuring the stability and reliability of the model inference service and meeting the service level agreement requirements. The dual-trigger mechanism can handle both predictable and continuous resource pressure (through capacity monitoring) and sudden, discrete optimization needs (through policy instructions). This enables the management of multi-level storage layers to possess both a global preventative perspective and the agility of local optimization, meeting the dynamic load and complex business scenarios in practical applications.

[0077] According to an embodiment of the present invention, selecting and invoking a hardware transmission link using a software abstraction interface includes: determining whether the initial storage layer and the intermediate storage layer are located on the same computing node based on the initial identifier of the initial storage layer and the intermediate identifier of the intermediate storage layer in the migration instruction; if they are located on the same computing node, invoking a point-to-point high-speed interconnect link using the software abstraction interface as the hardware transmission link; if they are not located on the same computing node, invoking a remote direct memory access link using the software abstraction interface as the hardware transmission link.

[0078] In embodiments of the present invention, the initial or intermediate identifier of the storage layer can be a logical tag used to refer to a specific instance of the storage layer in the system. It includes not only the layer type (such as G1), but also the device identifier (such as GPU serial number) and node identifier (such as hostname or IP), which are key metadata for determining the location relationship. The computing node can be a physically independent server unit.

[0079] Point-to-point high-speed interconnect links can be direct communication channels within the same computing node, enabling high bandwidth, low latency, and memory consistency between two processing units (such as GPU-1 and GPU-2), with data transmission not passing through the system's main memory. Remote direct memory access links can be network technologies that enable direct data transmission between the memory (or device memory) of different computing nodes, bypassing the operating system kernels at both ends and without CPU involvement. Remote memory read and write operations can be achieved through network protocols.

[0080] Figure 3 A schematic diagram illustrating an example of the migration process of key-value cache data blocks between different storage layers according to an embodiment of the present invention is shown.

[0081] like Figure 3 As shown, the computing nodes include computing node 1 and computing node 2. Computing node 1 contains multiple multi-level storage layers, including multi-level storage layer 11 and multi-level storage layer 12. Computing node 2 contains at least one multi-level storage layer 21.

[0082] Multi-level storage layer 11 contains four storage layers, which can be denoted as G1-11, G2-11, G3-11 and G4-11 respectively; multi-level storage layer 12 contains four storage layers, which can be denoted as G1-12, G2-12, G3-12 and G4-12 respectively; multi-level storage layer 21 contains four storage layers, which can be denoted as G1-21, G2-21, G3-21 and G4-21 respectively.

[0083] G1-12, G1-12, and G1-21 all contain core data, metadata, and other metadata. G1-12, G1-12, and G1-21 correspond to GPU memory; G2-11, G2-12, and G2-21 correspond to CPU memory; G3-11, G3-12, and G3-21 correspond to solid-state drives (SSDs); and G4-11, G4-12, and G4-21 correspond to remote storage.

[0084] For core data and metadata across different storage tiers within the same compute node, point-to-point high-speed interconnect links (such as PCIe) can be used as hardware transmission links to meet the high-performance requirements of data migration. For other metadata across different storage tiers, low-bandwidth message queues, such as Neural Autonomic Transport Systems (NATS), can be used for data transmission to allow for targeted data migration based on data availability. Data block migration between different compute nodes can utilize Remote Direct Memory Access links as hardware transmission links. Within the same compute node, for example, migration between G1-11 and G2-11 is via PCIe or NVLink (if supported by the GPU and CPU), between G2-11 and G3-11 is via the Non-Volatile Memory Express (NVMe) bus, and between G3-11 and G4-11 is via a network cable.

[0085] It should be noted that the number of computing nodes and multi-level storage layers mentioned above are just examples. Any number of computing nodes and multi-level storage layers can be used according to actual needs.

[0086] For example, software abstraction interfaces can achieve hardware adaptation and a unified interface. By encapsulating underlying hardware transmission technologies such as point-to-point high-speed interconnect links and remote direct memory access links, a unified interface is provided for the transmission of key-value cache data across devices or nodes, shielding hardware differences.

[0087] When transferring data between multiple GPUs within the same compute node (Worker), point-to-point high-speed interconnect links can be prioritized (if hardware supports it); otherwise, PCIe is used. For data transfer between GPUs from different Workers, remote direct memory access links are automatically enabled without manual configuration, achieving automatic selection of the transmission technology. A unified transmission interface is provided; simply passing the ID of the key-value cache data block and the identifier of the compute node triggers the transmission. The underlying layer automatically adapts to either point-to-point high-speed interconnect links or remote direct memory access links, without the compute node needing to concern itself with hardware details. GPU Direct Remote Memory Access (GPU Direct RDMA) technology is supported. When transferring data across compute nodes, data can be directly transferred from the initial GPU memory to the intermediate GPU memory without passing through the central processing unit's memory. When transferring data within the same Non-Uniform Memory Access (NUMA) node, the local network interface card is prioritized to reduce network latency.

[0088] According to embodiments of the present invention, through precise same-node or cross-node determination, the theoretically optimal hardware transmission path under the current physical topology is automatically matched for each data migration. For intra-node communication, unnecessary routing of data to low-speed networks is avoided; for cross-node communication, the most efficient RDMA technology is automatically enabled, thereby maximizing bandwidth and minimizing latency for cross-level, cross-device, and cross-node data transmission at the system level.

[0089] It is understandable that by completely encapsulating the complex hardware topology awareness and link selection logic in a software abstraction interface, the upper-layer scheduler only needs to issue the instruction "from A to B" without needing to know the physical location and connection method of A and B. This simplifies the development complexity of upper-layer applications. Furthermore, when the cluster hardware is upgraded or the topology changes, only the topology database and driver adaptation within the interface layer need to be updated. All upper-layer business logic can enjoy the performance improvement brought by the new hardware without any modification, which significantly improves the maintainability and scalability of the system.

[0090] According to an embodiment of the present invention, when the nodes are located on the same computing node, calling a point-to-point high-speed interconnect link using a software abstraction interface includes: determining whether there is an available point-to-point high-speed interconnect hardware link between the computing nodes based on the hardware connection topology between the computing nodes associated with the initial storage layer and the target storage layer; if there is, sending a first configuration instruction to the underlying driver through the software abstraction interface to configure and enable the point-to-point high-speed interconnect hardware link; if there is not, sending a second configuration instruction to the underlying driver through the software abstraction interface to call the peripheral component interconnect high-speed bus between the computing nodes.

[0091] In embodiments of the present invention, the hardware connection topology can indicate structured information about the physical or logical channels through which various computing devices are connected within the same computing node. This reflects whether point-to-point high-speed interconnect hardware links exist between devices, and how each device is connected via a shared bus (such as a peripheral component interconnect high-speed bus). The peripheral component interconnect high-speed bus can be a standardized, shared parallel communication bus system (i.e., PCIe bus) within the same computing node used to connect the CPU to various peripheral devices (such as GPUs, network cards, and memory controllers). Configuration instructions can be specific operation instructions generated by a software abstraction interface and sent to the underlying hardware driver.

[0092] For example, the built-in judgment logic of the software abstraction interface analyzes the query results. If the topology information indicates that there is a point-to-point high-speed interconnection hardware link between the two devices, and the link status is "available", then it is determined to "exist"; otherwise (e.g., there is no such link, or the link is disabled / faulty), then it is determined to "not exist".

[0093] In the presence of a high-speed link, a first configuration instruction is generated: the interface generates a configuration package containing the link type, device IDs at both ends, and transmission parameters. The interface calls the corresponding vendor-specific driver library and passes in the first configuration instruction; the driver configures and enables the driver library engine according to the instruction, establishing a direct memory access channel from the source GPU memory to the target GPU memory.

[0094] In the absence of a high-speed link, a second configuration instruction is generated: the interface generates a PCIe-based configuration packet, specifying transmission via a shared PCIe bus. The interface calls the operating system's standard PCIe driver and passes in the second configuration instruction; the driver configures the PCIe switch, establishing a peer-to-peer transmission channel via the PCIe bus.

[0095] According to embodiments of the present invention, by actively exploring hardware connection topology and prioritizing the use of existing point-to-point high-speed interconnect hardware links, it is possible to ensure that, under hardware conditions, data migration within a node can utilize the highest performance dedicated channel provided by the hardware. This allows the bandwidth of data flowing between GPUs or accelerators to reach the theoretical upper limit of direct device connection, minimizing latency. This provides crucial performance guarantees for strategies such as model parallelism or pipeline parallelism that require frequent data exchange within nodes.

[0096] It is understandable that by introducing "existence check" and "automatic fallback" mechanisms, upper-layer applications do not need to hardcode specific hardware configurations in their code. On servers equipped with high-speed interconnects, advanced functions can be automatically enabled; on servers equipped only with standard PCIe, they can also seamlessly degrade to use the standard bus for reliable operation. This reduces the software's dependence on different hardware configurations and allows the same system to be flexibly deployed in various hardware environments, significantly improving the versatility and ease of use of the solution.

[0097] According to an embodiment of the present invention, migrating key-value cache data blocks in an intermediate storage layer to a target storage layer in a multi-level storage layer includes: extracting physical memory description information from the metadata of the key-value cache data blocks, wherein the physical memory description information includes at least the starting memory address and memory layout information of the data in the initial storage layer; configuring a hardware transmission link based on the physical memory description information; reading key-value cache data in a non-contiguous memory region of the intermediate storage layer via the hardware transmission link according to the memory layout information; and writing the data in a distributed manner into a contiguous memory region of the target storage layer.

[0098] In embodiments of the present invention, metadata can refer to a set of information used to accurately locate and describe the distribution of data volumes in physical memory. Physical memory descriptors can be a set of parameters extracted from metadata to guide hardware in performing direct memory access. The starting memory address (base_address) can be a pointer to the starting position of the data block in the physical memory space of the source storage layer (intermediate storage layer). Memory layout information can be stride information (including memory alignment parameters).

[0099] The hardware transmission link can be an established and configured physical data transmission channel. In embodiments of the present invention, the hardware transmission link is configured to support a clustered-distributed operation mode. Reading can be a hardware-supported transmission mode that allows data to be read from multiple non-contiguous memory regions at once and logically aggregated into a continuous data stream within the transport layer (clustered read). Distributed writing can be a hardware transmission mode corresponding to clustered read, allowing the received continuous data stream to be written to multiple contiguous memory regions at once. A contiguous memory region can be a pre-allocated, physically contiguous block of memory space in the target storage layer (target storage layer) specifically used to receive migrated data.

[0100] For example, metadata can be synchronized across nodes. Key-value cached data blocks contain physical memory addresses and stride information for aggregation-dispersion operations during point-to-point high-speed interconnect links and remote direct memory access links (ensuring remote nodes can correctly read contiguous data segments of blocks); serialized remote layout metadata can include base offsets and physical memory descriptors, such as address, size, and device ID, for cross-node layout reconstruction, avoiding misalignment of remote data access.

[0101] For example, after the hardware transmission link is configured, the migration engine can be used to trigger the start of the transmission. During the aggregated read phase: the hardware transmission engine (such as an RDMA network card) automatically initiates a series of memory read operations based on the descriptors. It can read data of lengths len1, len2, ..., lenN from addr1, addr2, ..., addrN in parallel or pipelined manner. This data is integrated into an ordered data stream within the transmission engine or at the network layer. During the continuous write phase: this ordered data stream is continuously sent to the target end, and the target end's hardware directly writes it into a contiguous memory area, completing the physical migration and logical reorganization of the data.

[0102] According to embodiments of the present invention, by utilizing the hardware's native aggregated read capability, the overhead of multiple software copies and splicing caused by non-contiguous memory storage at the data source end is avoided. The hardware completes data collection and streaming at the transmission layer in one go, enabling high bandwidth and low latency for data migration from complex, fragmented memory layouts, thus improving the efficiency of data transfer between storage layers. Since the migration engine can intelligently handle non-contiguous storage based on memory layout information, this means that when managing key-value cache data blocks in the intermediate storage layer, a more flexible and memory-efficient allocation strategy can be adopted without forcibly maintaining contiguous data storage to ensure migration performance. This simplifies the design of the memory allocator, improves memory utilization, and provides greater flexibility for storage system optimization.

[0103] According to an embodiment of the present invention, configuring a hardware transmission link based on physical memory description information includes: extracting the base addresses and lengths of multiple non-contiguous memory regions, and the total length and starting address of the target contiguous memory region from memory layout information to obtain a transmission parameter set; generating a remote direct memory access transmission descriptor based on the transmission parameter set, wherein the remote direct memory access transmission descriptor includes an instruction sequence for instructing the hardware transmission link to perform aggregated read operations from multiple non-contiguous memory regions and distributed write operations to the target contiguous memory region; and submitting the remote direct memory access transmission descriptor to the command queue of the hardware transmission link to obtain a configured hardware transmission link.

[0104] In embodiments of the present invention, the transfer parameter set can be a set of structured parameters parsed and extracted from memory layout information, which fully describes the source, destination, and data organization of the transfer, and serves as metadata for generating executable transfer instructions. The target contiguous memory region can be a pre-allocated memory storage space. The Remote Direct Memory Access (RDMA) transfer descriptor can be a standardized data structure used in hardware that supports RDMA, encapsulating all control information required for a complete data transfer operation, and may include operation type, source / destination address list, data length, operation flags, etc., such as a microprogram or instruction package directly parsed and executed by hardware.

[0105] Instruction sequences can be sets of commands encoded in a specific format that constitute the core content of a remote direct memory access transfer descriptor. Command queues can be dedicated memory areas on RDMA hardware used to store remote direct memory access transfer descriptors submitted by host software and awaiting asynchronous hardware execution.

[0106] For example, the software calls the RDMA programming library, uses the transfer parameter set to populate an array of hash table entries, each hash table entry array corresponding to a source memory block, containing its address and length; creates a send job request structure, i.e., a remote direct memory access transfer descriptor; and combines the send job request structure and its bound hash table entry array to form a complete remote direct memory access transfer descriptor containing a sequence of instructions.

[0107] The descriptor is submitted to the command queue. The hardware engine of the RDMA network card polls its command queue and finds a new remote direct memory access transfer descriptor. The hardware resolves the descriptor and identifies the task that needs to read data from multiple local memory blocks and write it to a single remote contiguous address via RDMA operation. The network card starts the engine and, without CPU intervention, reads data from the addresses of the contiguous data blocks sequentially, combines them into a data packet stream, and sends it to the target node through the network. The target node receives the data packet and, based on the target address information in the data packet, directly writes the data to the contiguous memory region starting at the remote address via direct memory access.

[0108] According to embodiments of the present invention, by encoding the complex operation of reading from multiple discontinuous sources and writing to a contiguous target into a single remote direct memory access transfer descriptor, and executing it once by RDMA hardware, the overhead of multiple software copies, splicing, and system calls required in traditional methods is avoided. The CPU is only responsible for constructing and submitting the descriptor, and all subsequent data transfer is completed independently by the network card, achieving true zero-copy transfer and CPU offloading.

[0109] According to an embodiment of the present invention, the key-value cache data block has a two-dimensional array structure. The data volume of the first dimension of the two-dimensional array structure is the number of layers of the neural network that generates the key-value cache data block, and the data volume of the second dimension is the product of the predefined page size and the internal dimension of the model.

[0110] In embodiments of the present invention, the two-dimensional array structure can be an organizational format used when key-value cache data blocks are physically arranged in memory or storage devices. A predefined page size system can be set for managing key-value cache data blocks, specifying the number of tokens that a logical data block can hold. The internal dimension of the model can be the feature length (hidden layer size) of the key or value vector corresponding to each attention head in a neural network.

[0111] For example, the data in the key-value cache data block can be divided into two parts: core data and metadata. Two transmission channels, such as RDMA and Transmission Control Protocol (TCP), can be used to reduce the pressure on a single transmission channel. The core data consists of key and value tensor data generated during the large model inference process. It is directly used for subsequent token generation calculations. Its information format, storage method, and block layout are strongly bound. It includes: the core data of each key-value cache data block is organized in the form of a 2D array, with the dimension defined as: [num_layers][page_size×inner_dim].

[0112] num_layers is the number of Transformer layers in the corresponding large model; page_size can be the page size of the key-value cache data block (a pre-configured parameter, such as allocating a fixed number of token slots each time, for example, page_size=256 means that a single block can store key-value cache data of 256 tokens); inner_dim can be the internal dimension of the key-value cache data tensor (determined by the model structure, such as the hidden layer dimension of the key or value).

[0113] According to an embodiment of the present invention, the above method further includes: during the migration of key-value cache data blocks, generating a synchronization message containing updated metadata of the key-value cache data blocks, wherein the updated metadata includes the global identifier, storage location, data block size, and lifecycle status of the key-value cache data blocks; and sending the synchronization message to each computing node that has subscribed to metadata-related events through a message middleware based on a publish-subscribe pattern.

[0114] In embodiments of the present invention, updated metadata can be a subset of management information describing the latest state of a key-value cache data block, generated after a lifecycle event (such as creation, migration, or deletion) occurs. Synchronous messages are standardized communication units used to transmit updated metadata between different components of a distributed system. In a publish-subscribe model, the message sender (publisher) does not directly send messages to specific recipients, but instead categorizes and publishes messages to specific topics. Recipients (subscribers) interested in the topic receive a copy of the message, thus achieving one-to-many, decoupled communication. Message middleware can be used to reliably and asynchronously transmit messages between different applications or services.

[0115] For example, regardless of whether they are on the same computing node, the metadata of key-value cache data blocks is transmitted through a low-bandwidth channel to adapt to the small size and high-frequency synchronization characteristics of metadata.

[0116] Regarding the choice of transmission medium, within the same worker node, metadata is transmitted via local TCP sockets with latency as low as 10 microseconds. Between different worker nodes, metadata is transmitted via the event plane (based on the NATS messaging system, whose underlying protocol is TCP). NATS supports a publish-subscribe model, allowing metadata events to be broadcast in real time to all worker nodes that have subscribed to the topic, eliminating the need for point-to-point communication.

[0117] To optimize metadata content, a compact binary format can be used to encapsulate the metadata, which includes the following fields: key-value cache block ID (8 bytes), storage level (1 byte), location information (such as GPU ID or worker node IP address, 16 bytes), block size (4 bytes), and status (1 byte). The total length of a single metadata entry is only 30 bytes, thus significantly reducing the bandwidth requirements.

[0118] Regarding the synchronization timing, metadata is synchronized in real time along with the lifecycle of key-value cache data blocks. A new event is generated when the data is created, a location update event is generated when the data is migrated, and a deletion event is generated when the data is deleted. The Worker completes the local metadata update within 10ms after receiving the event, ensuring data consistency.

[0119] According to embodiments of the present invention, a publish-subscribe model is adopted, which completely decouples data producers (management nodes) from consumers (numerous worker nodes). When a new compute node is added, it only needs to subscribe to the corresponding topic to integrate into the cluster, without modifying the code of any publisher or other nodes. Similarly, changes to the management node will not affect the worker nodes. This loosely coupled architecture greatly improves the horizontal scalability of the entire inference cluster and simplifies system deployment and maintenance.

[0120] According to an embodiment of the present invention, the key-value cache data block is configured with a status flag indicating the current availability and data integrity; the method further includes: for key-value cache data blocks in the creation process, the status flag is uninitialized; for key-value cache data blocks in the writing process, the status flag is partially filled; for key-value cache data blocks whose writing process has been completed, the status flag is complete; and for key-value cache data blocks in the reclaimed state, the status flag is inactive.

[0121] In embodiments of the present invention, the status flag can be a metadata field associated with a key-value cache data block, used to characterize the current stage of the data block's lifecycle in the form of discrete enumerated values. An uninitialized state can indicate that the key-value cache data block has been created but has not yet begun receiving any valid data. A partially filled state can indicate that the key-value cache data block is receiving data writes but has not yet reached its predefined capacity. A complete state can indicate that the key-value cache data block has successfully received and stored all the valid data of its predefined capacity. An inactive state can indicate that the key-value cache data block has completed its service mission and has been marked by the system as a reclaimable resource.

[0122] For example, key-value cache data blocks can be divided into fixed-size blocks (e.g., 1024 Tokens / block), and the lifecycle of the data blocks can be managed through state marking: uninitialized state (Reset) → partially filled state (Partial) → complete state (Complete) → inactive state (Inactive). When a data block is created, it is allocated to the corresponding storage layer, and when it is reclaimed, resources are released and it is marked as inactive, waiting to be reused.

[0123] According to embodiments of the present invention, strict verification is performed before critical operations by using state flags, ensuring that data is only used for calculation in a complete state and will not be misused in an uninitialized, partially filled, or inactive state. This fundamentally prevents calculation errors, program crashes, or security vulnerabilities caused by reading uninitialized, incomplete, or released data.

[0124] According to an embodiment of the present invention, a status flag is used to determine the migration decision of a key-value cache data block, including: in response to an access request for the key-value cache data block, controlling the key-value cache data block with the status flag set to "complete" to be migrated out of the current storage layer; when allocating a storage layer for the current key-value cache data block, preferentially selecting the storage layer occupied by the key-value cache data block with the status flag set to "inactive".

[0125] In embodiments of the present invention, migration decision may be a process of determining whether, when, and how to migrate a key-value cache data block from an intermediate storage layer to a target storage layer, based on preset rules.

[0126] For example, when the state of a key-value cache data block changes (e.g., from Complete to Inactive), a state event is triggered, and the compute node updates the local block state based on the event to avoid accessing invalid data.

[0127] Metadata can include state metadata and ownership metadata of data blocks. For example, the lifecycle state metadata of a block can be used to record whether the block is currently available, whether it is full, and whether it can be shared; the ownership information of the data block: an uninitialized state indicates that ownership belongs to the InactivePool; a partially filled or fully filled state indicates that ownership belongs to the sequence creation thread; and a shared ownership state indicates that ownership belongs to the global block registry.

[0128] The device ID (device_ID) where the block resides can include, for example, GPU card number or CPU node ID, for cross-device scheduling; the storage medium type (memory_type) of the block can include VRAM / GPU memory, DRAM / CPU memory, SSD / disk, or remote memory, for matching hardware performance and data priority.

[0129] According to an embodiment of the present invention, the above method further includes: determining a target key-value cache data block whose access probability is greater than a preset threshold after processing the target text based on the historical access pattern information of the model inference request; and migrating the target key-value cache data block from the low-priority initial storage layer to the high-priority target storage layer during the current inference task's processing of the target text.

[0130] In embodiments of the present invention, historical access pattern information can be obtained by recording and analyzing the processing of model inference requests over a past period, and mining statistical patterns or machine learning models regarding the correlation between a specific context (target text) and subsequent accessed key-value cache data blocks. For example, it can indicate a causal or correlational relationship that "when X content is processed, Y data is likely to be needed next." The target text can be a specific segment, topic, or semantic context in the input sequence of the currently processed inference task. The key-value cache data block can be a specific data block that, based on analysis and threshold judgment, is identified as having an access probability greater than a preset threshold after the target text appears.

[0131] For example, hot data blocks are stored in storage layer G1, less hot data blocks are migrated to storage layer G2, cold data blocks are migrated to storage layer G3, and extremely cold data blocks are migrated to storage layer G4. When access to lower-level data is required, the migration can be triggered in advance (e.g., if it is predicted that a cold data block in G3 will be accessed within the next 100ms, it can be migrated to G2 in advance) to avoid inference waiting.

[0132] It is understood that the predictive migration strategy of data blocks in the embodiments of the present invention is a system behavior that actively and in advance moves the data from the low-priority initial storage layer to the high-priority target storage layer before the data is actually accessed by a request, thus realizing an optimization strategy that trades space for time.

[0133] For example, the resource utilization of each storage layer (such as G1 utilization, G2 utilization, and G3 remaining capacity) can be monitored in real time. When the G1 utilization is ≥90%, cold data migration is triggered, and when the G2 utilization is ≥80%, ultra-cold data migration to G3 or G4 is triggered.

[0134] For example, taking the large language model 70B as an example, the deployment parameters and process of this model are as follows:

[0135] The cluster configuration can include 4 GPU machines, each equipped with 4×200 Gbps RDMA NIC, 2TB CPU DRAM, and 8TB SSD. G4 uses distributed object storage. The storage tier configuration can include G1 (80GB of video memory per GPU), G2 (2TB of CPU DRAM per machine), G3 (8TB of SSD per machine), and G4 (total capacity of 100TB of distributed storage).

[0136] The framework can integrate key-value cache data block module processing device, transport library, and time plane; service level target settings can include: first token generation time P90 = 8 seconds (that is, for 90% of user requests, the generation time of the first output token does not exceed 8 seconds), and token generation time P90 = 30 milliseconds (that is, for 90% of user requests, the generation interval between two adjacent output tokens does not exceed 30 milliseconds).

[0137] The deployment and operation process can include user-initiated inference requests being distributed to various worker nodes via a load balancer. Within each worker node, the key-value cache block processing unit assesses the frequency of key-value cache data and allocates it to the corresponding storage tier. Within the same worker node, multiple graphics processors transmit key-value cache data via high-speed interconnects; between different worker nodes, data is transmitted using remote direct memory access technology. Metadata is synchronized in real-time via the event plane. When the remaining space in the first storage tier (GPU memory) is insufficient, the system automatically migrates infrequently accessed cold data to the third storage tier (local solid-state drive); when subsequent inference requires access to this data, it is migrated back to the first storage tier, thus ensuring the continuity and low latency of the inference process.

[0138] In one feasible embodiment, the embodiments of the present invention are adapted to large multimodal models such as text, image, and voice. The feature caches of different modalities (similar to text KV-cache) are included in the G1-G4 level management. For example, the image feature cache is stored in G3 (accessed less frequently than text), and the voice feature cache is stored in G4 (accessed least frequently). Cross-modal feature data transmission is achieved through point-to-point high-speed interconnection links. The metadata synchronization mechanism remains unchanged, and only the "modal type" field in the metadata needs to be extended (e.g., 1 = text, 2 = image, 3 = voice).

[0139] In edge-cloud collaborative inference, edge nodes (such as edge servers) are deployed with G1 and G2, while the cloud is deployed with G3 and G4. Cold data from edge nodes is migrated to cloud storage on G4. When needed, data is migrated back to the edge via a 5G / 6G network (compatible with RDMA over 5G technology) to solve the problem of limited storage capacity at edge nodes. At the same time, the Event Plane is used to synchronize metadata between the edge and the cloud, ensuring consistency in collaborative inference.

[0140] For heterogeneous hardware clusters, this solution is adapted to GPU, CPU, and FPGA heterogeneous clusters. The FPGA is used as a dedicated transmission acceleration unit, and the verification and encoding of NVLink / RDMA data transmission are optimized to further reduce transmission latency. At the same time, the external SSD of the FPGA can be used as G3 extended storage. KV-cache cold data is migrated to the external SSD of the FPGA to improve the storage and transmission coordination efficiency of heterogeneous clusters. Only the FPGA storage level adaptation logic needs to be added to the key-value cache data processing device in model inference, and the other modules do not need to be modified.

[0141] Based on the above-described method for processing key-value cached data, this invention also provides a device for processing key-value cached data in model inference. The following will combine... Figure 4 The device is described in detail.

[0142] Figure 4 A structural block diagram of a device for processing key-value cache data in model inference according to an embodiment of the present invention is shown.

[0143] like Figure 4 As shown, the key-value cache data processing device 400 in this embodiment includes a storage module 410, a selection module 420, and a migration module 430.

[0144] Storage module 410 is used to allocate key-value cache data blocks to the initial storage layer in a multi-level storage layer for storage based on the access priority of the key-value cache data blocks of the task to be inferred. The access priority is determined by the access frequency of the key-value cache data blocks and the cost information for storing the key-value cache data blocks. In one embodiment, storage module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0145] Selection module 420 is used to select and invoke a hardware transmission link to migrate key-value cache data blocks from the initial storage layer to the intermediate storage layer based on the heat assessment result of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, using a software abstraction interface. The heat assessment result indicates the migration direction of the key-value cache data blocks. In one embodiment, selection module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0146] The migration module 430 is configured to, in response to an access request for a key-value cache data block, migrate the key-value cache data block in the intermediate storage layer to the target storage layer in the multi-level storage layer, so as to perform decoding based on the key-value cache data block in the target storage layer and generate an inference result. In one embodiment, the migration module 430 may be used to perform the operation S230 described above, which will not be repeated here.

[0147] According to an embodiment of the present invention, the multi-level storage layer includes: a first storage layer serving as graphics processor memory, used to store key-value cache data blocks with an access frequency greater than a first threshold; a second storage layer serving as central processing unit memory, used to store key-value cache data blocks with an access frequency less than or equal to the first threshold and greater than a second threshold; a third storage layer serving as a local solid-state drive, used to store key-value cache data blocks with an access frequency less than or equal to the second threshold and greater than a third threshold; and a fourth storage layer serving as remote object storage, used to store key-value cache data blocks with an access frequency less than or equal to the third threshold.

[0148] According to an embodiment of the present invention, the storage module 410 includes: an acquisition submodule, a determination submodule, and a selection submodule. The acquisition submodule is used to acquire cost information of each storage layer in a multi-level storage layer, the cost information including unit capacity cost information and access latency information of the storage medium in each storage layer; the determination submodule is used to determine the matching degree between the access priority and the cost information of each storage layer, obtaining multiple matching degrees; the selection submodule is used to select the storage layer corresponding to the peak matching degree from the multiple matching degrees as the initial storage layer, so as to store the key-value cache data block in the initial storage layer.

[0149] According to an embodiment of the present invention, the above-mentioned device 400 further includes: a mapping module, a utilization rate determination module, a query module, and a weighting module. The mapping module is used to map access priorities to target values ​​within a preset range as base popularity values; the utilization rate determination module is used to determine the ratio between the used capacity of each storage layer and the total capacity in a multi-level storage layer as the capacity utilization rate of each storage layer; the query module is used to query a type-coefficient mapping table according to the trigger signal type in the migration instruction to obtain the adjustment coefficient corresponding to the trigger signal type; the weighting module is used to perform a weighted summation of the base popularity value, the cost information of each storage layer, the capacity utilization rate, and the adjustment coefficient using preset weights to obtain a popularity evaluation result.

[0150] According to an embodiment of the present invention, the selection module 420 includes a response submodule, a detection submodule, and a receiving submodule. The response submodule is configured to migrate a key-value cache data block stored in the initial storage layer to an intermediate storage layer in response to one of the following events: the detection submodule detects that the remaining capacity of the initial storage layer is less than a capacity threshold; the receiving submodule receives an access request for a target key-value cache data block in the initial storage layer.

[0151] According to an embodiment of the present invention, the selection module 420 further includes: a node determination submodule, a calling submodule, and a link calling submodule. The node determination submodule is used to determine whether the initial storage layer and the intermediate storage layer are located on the same computing node based on the initial identifier of the initial storage layer and the intermediate identifier of the intermediate storage layer in the migration instruction; the calling submodule is used to call a point-to-point high-speed interconnect link using a software abstraction interface as a hardware transmission link when they are located on the same computing node; the link calling submodule is used to call a remote direct memory access link using a software abstraction interface as a hardware transmission link when they are not located on the same computing node.

[0152] According to an embodiment of the present invention, the calling submodule includes: a link determination unit, a sending unit, and an instruction sending unit. The link determination unit is used to determine whether an available point-to-point high-speed interconnect hardware link exists between computing nodes based on the hardware connection topology between the computing nodes associated with the initial storage layer and the target storage layer, respectively. The sending unit is used to, if the link exists, send a first configuration instruction to the underlying driver through a software abstraction interface to configure and enable the point-to-point high-speed interconnect hardware link. The instruction sending unit is used to, if the link does not exist, send a second configuration instruction to the underlying driver through a software abstraction interface to invoke the peripheral component interconnect high-speed bus between computing nodes.

[0153] According to an embodiment of the present invention, the migration module 430 includes an extraction submodule and a configuration submodule. The extraction submodule is used to extract physical memory description information from the metadata of the key-value cache data block, wherein the physical memory description information includes at least the starting memory address and memory layout information of the data in the initial storage layer; the configuration submodule is used to configure a hardware transmission link based on the physical memory description information, and according to the memory layout information, read the key-value cache data in the non-contiguous memory region of the intermediate storage layer via the hardware transmission link, and write it in a distributed manner to the contiguous memory region of the target storage layer.

[0154] According to an embodiment of the present invention, the configuration submodule includes: an address extraction unit, a generation unit, and a submission unit. The address extraction unit is used to extract the base addresses and lengths of multiple non-contiguous memory regions, and the total length and starting address of the target contiguous memory region, from memory layout information to obtain a transmission parameter set. The generation unit is used to generate a remote direct memory access (RDA) transfer descriptor based on the transfer parameter set, wherein the RDA transfer descriptor includes a sequence of instructions instructing the hardware transmission link to perform aggregated read operations from multiple non-contiguous memory regions and distributed write operations to the target contiguous memory region. The submission unit is used to submit the RDA transfer descriptor to the command queue of the hardware transmission link to obtain a configured hardware transmission link.

[0155] According to an embodiment of the present invention, the key-value cache data block has a two-dimensional array structure. The data volume of the first dimension of the two-dimensional array structure is the number of layers of the neural network that generates the key-value cache data block, and the data volume of the second dimension is the product of the predefined page size and the internal dimension of the model.

[0156] According to an embodiment of the present invention, the above-described apparatus 400 further includes a message generation module and a message sending module. The message generation module is used to generate a synchronization message containing updated metadata of the key-value cache data block during the migration of the key-value cache data block, wherein the updated metadata includes the global identifier, storage location, data block size, and lifecycle status of the key-value cache data block; the message sending module is used to send the synchronization message to each computing node that has subscribed to metadata-related events through a message middleware based on a publish-subscribe pattern.

[0157] According to an embodiment of the present invention, the key-value cache data block is configured with a status flag indicating its current availability and data integrity; the device further includes: a first flag module, a second flag module, a third flag module, and a fourth flag module. The first flag module is used to mark the status of a key-value cache data block in the creation process as uninitialized; the second flag module is used to mark the status of a key-value cache data block in the writing process as partially filled; the third flag module is used to mark the status of a key-value cache data block whose writing process has been completed as complete; and the fourth flag module is used to mark the status of a key-value cache data block in the reclaimed state as inactive.

[0158] According to an embodiment of the present invention, a status flag is used to determine the migration decision of a key-value cache data block, including: in response to an access request for the key-value cache data block, controlling the key-value cache data block with the status flag set to "complete" to be migrated out of the current storage layer; when allocating a storage layer for the current key-value cache data block, preferentially selecting the storage layer occupied by the key-value cache data block with the status flag set to "inactive".

[0159] According to an embodiment of the present invention, the above-described apparatus further includes: a data block determination module and a data block migration module. The data block determination module is used to determine a target key-value cache data block whose access probability after processing the target text is greater than a preset threshold based on historical access pattern information of the model inference request; the data block migration module is used to migrate the target key-value cache data block from a low-priority initial storage layer to a high-priority target storage layer during the current inference task's processing of the target text.

[0160] According to embodiments of the present invention, any plurality of modules among the storage module 410, selection module 420, and migration module 430 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the storage module 410, selection module 420, and migration module 430 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the storage module 410, selection module 420, and migration module 430 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0161] Figure 5 A block diagram of an electronic device suitable for implementing a key-value cache data processing method according to an embodiment of the present invention is shown.

[0162] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0163] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0164] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0165] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0166] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0167] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the key-value cache data processing method provided in the embodiments of the present invention.

[0168] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0169] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0170] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0171] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing node, partially on the user's device, partially on a remote computing node, or entirely on a remote computing node or server. In cases involving remote computing nodes, the remote computing node can be connected to the user's computing node via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing node (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0174] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for processing key-value cached data, characterized in that, The method includes: Based on the access priority of the key-value cache data block of the task to be inferred, the key-value cache data block is allocated to the initial storage layer in the multi-level storage layer for storage. The access priority is determined by the access frequency of the key-value cache data block and the cost information of storing the key-value cache data block. Based on the heat assessment results of the key-value cache data blocks and the positional relationship between the initial storage layer and the intermediate storage layer in the multi-level storage layer, the key-value cache data blocks in the initial storage layer are migrated to the intermediate storage layer by selecting and calling the hardware transmission link using a software abstraction interface. The heat assessment results indicate the migration direction of the key-value cache data blocks. Elements affecting the heat assessment results include: the access priority of the data blocks, the cost information of the storage layer, the trigger signal type in the migration instruction, and the capacity utilization rate of the storage layer. In response to an access request for the key-value cache data block, the key-value cache data block in the intermediate storage layer is migrated to the target storage layer in the multi-level storage layer, including: extracting physical memory description information from the metadata of the key-value cache data block to guide the hardware to perform direct memory access, wherein the physical memory description information includes at least the starting memory address and memory layout information of the data in the initial storage layer; configuring a hardware transmission link that supports aggregated read and distributed write operation modes based on the physical memory description information; reading key-value cache data in the non-contiguous memory area of ​​the intermediate storage layer via the hardware transmission link according to the memory layout information; logically aggregating the read data into a continuous data stream within the transmission layer; and distributively writing the data into a continuous memory area in the target storage layer for receiving the migrated data; and decoding based on the key-value cache data block in the target storage layer to generate an inference result.

2. The method according to claim 1, characterized in that, The multi-level storage layer includes: As the first storage layer of the graphics processor's video memory, it is used to store key-value cache data blocks whose access frequency is greater than a first threshold. As a second storage layer of the central processing unit memory, it is used to store key-value cache data blocks whose access frequency is less than or equal to the first threshold and greater than the second threshold. As the third storage layer of the local solid-state drive, it is used to store key-value cache data blocks whose access frequency is less than or equal to the second threshold and greater than the third threshold; As a fourth storage layer for remote object storage, it is used to store key-value cache data blocks whose access frequency is less than or equal to the third threshold.

3. The method according to claim 1 or 2, characterized in that, Based on the access priority of the key-value cache data block of the task to be inferred, the key-value cache data block is allocated to the initial storage layer in the multi-level storage layer for storage, including: Obtain cost information for each storage layer in the multi-level storage layer, the cost information including unit capacity cost information and access latency information of the storage medium in each storage layer; Determine the matching degree between the access priority and the cost information of each storage layer to obtain multiple matching degrees; The storage layer corresponding to the peak matching degree is selected from the multiple matching degrees and used as the initial storage layer to store the key-value cache data block in the initial storage layer.

4. The method according to claim 3, characterized in that, The method further includes: The access priority is mapped to a target value within a preset range, which is used as the base popularity value; The ratio between the used capacity of each storage layer and the total capacity in the multi-level storage layer is determined as the capacity utilization rate of each storage layer. Based on the trigger signal type in the migration instruction, query the type-coefficient mapping table to obtain the adjustment coefficient corresponding to the trigger signal type; The heat assessment result is obtained by weighting and summing the base heat value, the cost information of each storage layer, the capacity utilization rate and the adjustment coefficient using preset weights.

5. The method according to claim 1, characterized in that, Migrating key-value cache data blocks in the initial storage layer to the intermediate storage layer includes: In response to one of the following events, the key-value cache data blocks stored in the initial storage layer are migrated to the intermediate storage layer: The detection of the capacity status of the initial storage layer indicates that the remaining capacity of the initial storage layer is less than the capacity threshold. An access request was received for the target key-value cache data block in the initial storage layer.

6. The method according to claim 1, characterized in that, Using a software abstraction interface, select and invoke the hardware transmission link, including: Based on the initial identifier of the initial storage layer and the intermediate identifier of the intermediate storage layer in the migration instruction, determine whether the initial storage layer and the intermediate storage layer are located on the same computing node; When located on the same computing node, the software abstraction interface is used to call the point-to-point high-speed interconnect link as the hardware transmission link. When the nodes are not located on the same computing node, the software abstraction interface is used to call the remote direct memory access link as the hardware transmission link.

7. The method according to claim 6, characterized in that, When located on the same computing node, using the software abstraction interface to invoke the point-to-point high-speed interconnect link includes: Based on the hardware connection topology between the computing nodes associated with the initial storage layer and the target storage layer, determine whether there are available point-to-point high-speed interconnect hardware links between the computing nodes; If present, a first configuration instruction is sent to the underlying driver through the software abstraction interface to configure and enable the point-to-point high-speed interconnect hardware link; In the absence of such a configuration, a second configuration instruction is sent to the underlying driver through the software abstraction interface to invoke the high-speed bus for interconnecting peripheral components between the computing nodes.

8. The method according to claim 1, characterized in that, Based on the physical memory description information, configure a hardware transmission link that supports both aggregated read and distributed write operation modes, including: The base addresses and lengths of multiple non-contiguous memory regions, as well as the total length and starting address of the target contiguous memory region, are extracted from the memory layout information to obtain the transmission parameter set; Based on the set of transmission parameters, a remote direct memory access transmission descriptor is generated, wherein the remote direct memory access transmission descriptor includes a sequence of instructions for instructing the hardware transmission link to perform aggregated read operations from multiple non-contiguous memory regions and to perform distributed write operations to the target contiguous memory region; The remote direct memory access transfer descriptor is submitted to the command queue of the hardware transfer link to configure the hardware transfer link.

9. The method according to claim 1, characterized in that, The key-value cache data block has a two-dimensional array structure. The data volume of the first dimension of the two-dimensional array structure is the number of layers of the neural network that generates the key-value cache data block, and the data volume of the second dimension is the product of the predefined page size and the internal dimension of the model.

10. The method according to claim 1, characterized in that, The method further includes: During the migration of the key-value cache data block, a synchronization message containing updated metadata of the key-value cache data block is generated, wherein the updated metadata includes the global identifier, storage location, data block size, and lifecycle status of the key-value cache data block; The synchronization message is sent to each computing node that has subscribed to metadata-related events via a message middleware based on a publish-subscribe pattern.

11. The method according to claim 1, characterized in that, The key-value cache data block is configured with a status flag indicating its current availability and data integrity; the method further includes: For the key-value cache data block that is in the process of being created, the status is marked as uninitialized; For key-value cache data blocks that are being written, the status is marked as partially filled. For key-value cache data blocks whose write process has been completed, the status is marked as complete. For key-value cache data blocks that have been reclaimed, the status is marked as inactive.

12. The method according to claim 11, characterized in that, The status flag is used to determine the migration decision of the key-value cache data block, including: In response to an access request for the key-value cache data block, the key-value cache data block whose status is marked as complete is migrated out of the current storage layer; When allocating storage layers for the current key-value cache data block, priority is given to the storage layers occupied by key-value cache data blocks whose status is marked as inactive.

13. The method according to any one of claims 1 to 2, characterized in that, The method further includes: Based on the historical access pattern information of the model inference request, the target key-value cache data block with a probability of being accessed after processing the target text is determined to be greater than a preset threshold. During the current inference task's processing of the target text, the target key-value cache data block is migrated from the low-priority initial storage layer to the high-priority target storage layer.

14. An electronic device comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

Images