Tensor cache transmission method and device, equipment and storage medium

Abstract

The application discloses a tensor cache transmission method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: receiving a cache transmission request, the cache transmission request being used for requesting to transmit a tensor cache, the tensor cache being used for caching at least one of K tensors and V tensors generated in an LLM inference process; and transmitting the tensor cache from the host memory of a CPU to the device memory of an acceleration device through a transfer mode, or transmitting the tensor cache from the device memory of the acceleration device to the host memory of the CPU through the transfer mode; wherein the acceleration device is used for accelerating the inference process of the LLM. The application can avoid the tensor cache transmission risk introduced by directly dereferencing the CPU host pointer.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for transmitting tensor buffers. Background Technology

[0002] During Large Language Model (LLM) inference, a key (K) and value (V) cache is maintained. The KV cache is used to store the K tensor / V tensor of historical tokens, avoiding the recalculation of historical context each time a new token is generated.

[0003] As context length, the number of concurrent requests, and model size increase, the memory consumed by the key-value cache grows rapidly. To reduce memory pressure, inference frameworks typically employ cache offloading techniques, offloading a portion of the key-value cache from the device memory of the LLM's acceleration device to the host memory of the Central Processing Unit (CPU), and then loading it back into device memory when needed by subsequent requests. In related technologies, by exposing the CPU-pinned host memory pointer to the device kernel of the acceleration device, the device kernel can directly access the CPU-pinned host memory pointer, thereby achieving the offloading and reloading of the key-value cache.

[0004] The above method of transferring KV cache between the CPU's host memory and the acceleration device's device memory carries certain transmission risks because it requires the device kernel to directly dereference the CPU host pointer. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for transferring tensor caches, which can avoid the risks of tensor cache transfer introduced by directly dereferencing the CPU host pointer. The technical solution includes the following:

[0006] According to one aspect of this application, a method for transferring a tensor buffer is provided, the method comprising the following steps.

[0007] Receive a cache transfer request, the cache transfer request being used to request the transfer of the tensor cache, the tensor cache being used to cache at least one of the K tensor and V tensor generated during the LLM inference process; The tensor cache is transferred from the CPU's host memory to the acceleration device's device memory via a relay; or, The tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay method; The acceleration device is used to accelerate the inference process of the LLM.

[0008] According to another aspect of this application, a tensor buffer transmission apparatus is provided, the apparatus comprising the following modules.

[0009] A receiving module is configured to receive a cache transmission request, wherein the cache transmission request is used to request the transmission of the tensor cache, wherein the tensor cache is used to cache at least one of the K tensor and V tensor generated during the LLM inference process; A transmission module is used to transfer the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay method; or, The tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay method; The acceleration device is used to accelerate the inference process of the LLM.

[0010] In an optional design, the transmission module is used for: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache into the device cache in the device memory used to store the tensor cache.

[0011] In an optional design, the transmission module is used for: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

[0012] In an optional design, the cache transfer request includes a set of device indexes, which indicates the location of a device cache in the device memory used to store the tensor cache; the transfer module is configured to: When the indices in the device index set are consecutive and the device cache is in a layer-first layout, the tensor cache in the temporary cache is written to the device cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the device index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache in the temporary cache is written into the device cache by the first operator of the acceleration device. The first operator is used to adapt the layout of the tensor cache to the layout of the device cache.

[0013] In an optional design, the cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; the transfer module is configured to: When the indices in the host index set are consecutive, the contiguous memory view corresponding to the indices in the host index set is obtained from the host cache through a selection operation or a slicing operation to obtain the tensor cache; When the indices in the host index set are not contiguous, the slots corresponding to the indices in the host index set are obtained from the host cache through an index selection operation to obtain the tensor cache.

[0014] In an optional design, the tensor cache accessed via the CPU's host memory includes linear tensors.

[0015] In an optional design, the transmission module is used for: The tensor cache is obtained through the device cache in the device memory used to store the tensor cache; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache to the host memory of the CPU.

[0016] In an optional design, the cache transfer request includes a set of device indexes, which indicates the location of a device cache in the device memory used to store the tensor cache; the transfer module is configured to: When the indices in the device index set are consecutive and the device cache is a layer-first layout, the slice corresponding to the index in the device index set is directly obtained from the device cache to obtain the tensor cache; If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache is obtained from the device cache by the second operator of the acceleration device, the second operator being used to adjust the layout of the data in the device cache.

[0017] In an alternative design, the tensor cache obtained through the device memory of the acceleration device includes linear tensors.

[0018] In an optional design, the transmission module is used for: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

[0019] In an optional design, the cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; the transfer module is configured to: Copy the tensor cache in the temporary cache to the page-locked memory of the CPU; When the indexes in the host index set are consecutive, the tensor cache in the page-locked memory of the CPU is written back to the host cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the host index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indexes in the host index set are not contiguous, the tensor cache in the page-locked memory of the CPU is written back to the slots corresponding to the respective indexes in the host index set through an index copy operation, so as to write the tensor cache back to the host cache.

[0020] In an optional design, the transmission module is used for: If the cache transfer request meets the first condition, the tensor cache is transferred from the host memory of the CPU to the device memory of the acceleration device via a relay. The first condition includes at least one of the following: the cache transfer request instructing the CPU's host memory to transfer the tensor cache to the acceleration device's device memory, and the transfer switch being enabled.

[0021] In an optional design, the transmission module is used for: If the cache transfer request meets the second condition, the tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay. The second condition includes at least one of the following: the cache transfer request instructing the tensor cache to be transferred from the device memory of the acceleration device to the host memory of the CPU; and the transfer switch being enabled.

[0022] In an optional design, the CPU's host memory includes CPU page-locked host memory.

[0023] In an optional design, the transmission module is used for: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is written into a pre-created intermediate memory region in the CPU; The tensor cache in the transit memory region is transferred to the device memory of the acceleration device.

[0024] In an optional design, the transmission module is used for: The tensor cache is obtained through the device memory of the acceleration device; The acquired tensor cache is transferred to a pre-created intermediate memory region in the CPU; The tensor cache in the intermediate memory region is read into the host memory of the CPU.

[0025] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the tensor buffer transfer method as described above.

[0026] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, the at least one program being loaded and executed by a processor to implement the tensor buffer transfer method as described above.

[0027] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the tensor buffer transfer method provided in various alternative implementations of the above aspects.

[0028] The beneficial effects of the technical solution provided in this application include at least the following: By transferring the tensor cache from the CPU's host memory to the accelerator's device memory, or vice versa, a method other than accessing the CPU's host pointer, is implemented to transfer the tensor cache between the CPU's host memory and the accelerator's device memory, thus avoiding the transmission risks introduced by directly dereferencing the CPU's host pointer. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the structure of a converter provided in an exemplary embodiment of this application; Figure 2 This is a schematic diagram of a transport tensor caching process provided in an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a tensor buffer transfer method provided in an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating another method for transferring tensor buffers provided in an exemplary embodiment of this application; Figure 5 This is a flowchart illustrating another method for transferring tensor buffers provided in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the system architecture provided in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating an integration method provided in an exemplary embodiment of this application; Figure 8 This is a routing C data flow graph provided in an exemplary embodiment of this application; Figure 9 This is a schematic diagram of a diagnostic mode provided in an exemplary embodiment of this application; Figure 10 This is a schematic diagram of the transmission process of a KV cache provided in an exemplary embodiment of this application; Figure 11 This is a schematic diagram of the structure of a tensor buffer transmission device provided in an exemplary embodiment of this application; Figure 12 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] LLM is a deep learning model based on the Transformer architecture, possessing billions to hundreds of billions of parameters. Trained on massive amounts of text data, it learns the rules and knowledge of language. Its core mechanism is self-attention, which enables it to understand contextual semantics and predict and generate text word by word, thus enabling various language capabilities such as dialogue, question answering, writing, translation, and code generation.

[0034] For example, Figure 1 This is a schematic diagram of the converter structure provided in an exemplary embodiment of this application. Figure 1 As shown, the converter 101 includes an encoding network 102 and a decoding network 103. By inputting input information into the encoding network 102, the feature extraction result of the encoding network 102 for the input information can be obtained, i.e., the encoded information. By inputting the encoded information into the decoding network 103, the output information of the decoding network 103 can be obtained, i.e., the prediction information that matches the input information. The encoding network 102 and the decoding network 103 are N-layer structures. The encoding network 102 is a cascaded structure of N encoders, and the decoding network 103 is a cascaded structure of N decoders. The structure of each layer in the encoding network 102 is consistent, the structure of each layer in the decoding network 103 is consistent, and the structure of each layer in the encoding network 102 is similar to that in the decoding network 103.

[0035] Continue to refer to Figure 1 Each layer (encoder) of the encoding network 102 typically includes a multi-head self-attention module, i.e. Figure 1 The left-hand encoder structure contains "self-attention" and a fully connected feedforward network (also known as a feedforward network, FFN)... Figure 1 The "feedforward full connection" in the encoder structure on the left side of the middle.

[0036] Continue to refer to Figure 1 Each layer (decoder) of the decoding network 103 typically includes a mask multi-head self-attention module (which can be considered a type of multi-head self-attention module), i.e. Figure 1The "self-attention" module at the bottom of the decoder structure on the right side. This self-attention module of a cross encoder and decoder (also known as a cross self-attention module, which can be considered a multi-head self-attention module) is... Figure 1 The self-attention mechanism in the middle of the right-hand decoder structure, along with a feedforward fully connected module. Figure 1 The upper part of the decoder structure on the right side is a "feedforward fully connected".

[0037] In this network, the multi-head self-attention module of the encoding network 102 is used to obtain the weight relationship of each word in the input text compared to other words in the input text. The feedforward fully connected module of the encoding network 102 is used to perform non-linear transformation on the input features. The masked multi-head self-attention module of the decoding network 103 has a similar function to the multi-head self-attention module of the encoding network 102, except that it is also used to prevent the decoding network 103 from obtaining the prediction results corresponding to the words following that word in the input text when generating a prediction result that matches a certain word in the input text (during training, this is done). Figure 1 The lower position in the decoder structure on the right side represents the prediction result corresponding to the input text. The cross-self-attention module of the decoding network 103 functions similarly to the multi-head self-attention module of the encoding network 102, except that its input consists of the output information of the previous module in the decoding network 103 and the output information of the last layer in the encoding network 102. The feedforward fully connected module of the decoding network 103 functions similarly to the feedforward fully connected module of the encoding network 102.

[0038] In addition, continue to refer to Figure 1 Each of the modules (multi-head self-attention module, feedforward fully connected module) in the encoding network 102 and decoding network 103 of the converter 101 is provided with a residual connection and a layer normalization (LayerNorm) layer (i.e. Figure 1 The model employs residual and normalization (Add & Norm) layers. Residual connections can be viewed as structures that allow the output of a module to serve as the input to a subsequent, non-adjacent module, reducing model complexity and preventing gradient vanishing. Normalization layers are used to normalize the input information, such as through standardization. Both residual connections and normalization layers stabilize the model's training. During the training of transformer 101, the computer acquires a large amount of textual data for pre-training, enabling transformer 101 to achieve good generalization performance across different domains of text.

[0039] In LLM inference processes, such as autoregressive LLM, the attention computation for the transformer maintains a KV cache. The KV cache stores the K tensor / V tensor of the LLM's historical tokens, avoiding redundant computation of historical context each time a new token is generated. The K tensor can be understood as the feature label of the token, used to store its semantic features. The K tensor is used to calculate the attention weights, helping the LLM filter out other tokens that need attention for the current token. The V tensor can be understood as the actual information carrier of the token, used to store its semantic content. The V tensor is used to perform a weighted sum based on the attention weights to obtain the output of the LLM's attention layer.

[0040] As context length, the number of concurrent requests, and model size increase, the memory consumed by the key-value cache grows rapidly. To reduce memory pressure, inference frameworks corresponding to LLM typically employ cache offloading techniques, offloading part of the key-value cache from the device memory of the LLM acceleration device to the CPU's host memory, and then loading it back into device memory when needed by subsequent requests. In related technologies, by exposing the CPU pinned host memory pointer to the device kernel of the acceleration device, the device kernel can directly access the CPU pinned host memory pointer, thereby achieving the offloading and reloading of the key-value cache.

[0041] The above method of transferring KV cache between the CPU's host memory and the device memory of the acceleration device has transmission risks such as data inconsistency risk, address space risk, synchronization risk, and abnormal access risk, because the device kernel needs to directly dereference the CPU host pointer.

[0042] The method provided in this application uses a staged transfer as a secure and verifiable path. Specifically, it transfers the tensor cache (KV cache) from the CPU's host memory to the accelerator's device memory, or vice versa, via a relay mechanism. This allows for the transfer of the tensor cache between the CPU's host memory and the accelerator's device memory through methods other than accessing the CPU's host pointer, thereby avoiding the transfer risks introduced by directly dereferencing the CPU's host pointer.

[0043] Figure 2 This is a schematic diagram of a transport tensor buffering process provided in an exemplary embodiment of this application. For example... Figure 2As shown, computer device 200 includes CPU 201 and acceleration device 202. An LLM-based inference service runs in computer device 200, and acceleration device 202 is used to accelerate the LLM inference process. Exemplarily, acceleration device 202 includes at least one of a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), and other devices besides GPU and NPU. During LLM inference, a tensor cache 203 is generated, which is used to cache at least one of K tensors and V tensors generated during LLM inference. In some embodiments, tensor cache 203 includes a KV cache.

[0044] When it is necessary to transfer tensor cache 203, computer device 200 receives a cache transfer request, which requests the transfer of tensor cache 203. Computer device 200 transfers tensor cache 203 from host memory 2011 of CPU 201 to device memory 2021 of acceleration device 202 via a relay; or, computer device 200 transfers tensor cache 203 from device memory 2021 of acceleration device 202 to host memory 2011 of CPU 201 via a relay. For example, when a cache transfer request indicates that tensor cache 203 be transferred from the host memory 2011 of CPU 201 to the device memory 2021 of acceleration device 202, computer device 200 transfers tensor cache 203 from the host memory 2011 of CPU 201 to the device memory 2021 of acceleration device 202 via a relay; when a cache transfer request indicates that tensor cache 203 be transferred from the device memory 2021 of acceleration device 202 to the host memory 2011 of CPU 201, computer device 200 transfers tensor cache 203 from the device memory 2021 of acceleration device 202 to the host memory 2011 of CPU 201 via a relay. In some embodiments, the host memory 2011 of CPU 201 includes CPU page-locked host memory 2011. The relay method is other than directly transferring the tensor cache 203 between the host memory 2011 of CPU 201 and the device memory 2021 of acceleration device 202. For example, the tensor cache 203 can be transferred between the host memory 2011 of CPU 201 and the device memory 2021 of acceleration device 202 through the relay within CPU 201, or the tensor cache 203 can be transferred between the host memory 2011 of CPU 201 and the device memory 2021 of acceleration device 202 through the relay within acceleration device 202.

[0045] Optionally, for the process of transferring tensor cache 203 from the host memory 2011 of CPU 201 to the device memory 2021 of acceleration device 202: computer device 200 obtains tensor cache 203 through the host memory 2011 of CPU 201, copies the obtained tensor cache 203 to the temporary cache of acceleration device 202, and then writes the tensor cache 203 in the temporary cache into the device memory 2021 of acceleration device 202 for storing tensor cache 203. Alternatively, for the process of transferring tensor cache 203 from the device memory 2021 of acceleration device 202 to the host memory 2011 of CPU 201: computer device 200 obtains tensor cache 203 through the device cache in the device memory 2021 of acceleration device 202 for storing tensor cache 203, copies the obtained tensor cache 203 to the temporary cache of acceleration device 202, and then writes the tensor cache 203 in the temporary cache into the host memory 2011 of CPU 201. The temporary cache of the acceleration device 202 can be understood as a cache used to bridge the host memory 2011 of the CPU 201 and the device memory 2021 of the acceleration device 202, or a cache used to relay data transferred between the host memory 2011 of the CPU 201 and the device memory 2021 of the acceleration device 202.

[0046] By transferring the tensor cache from the CPU's host memory to the accelerator's device memory, or vice versa, a method other than accessing the CPU's host pointer, is implemented to transfer the tensor cache between the CPU's host memory and the accelerator's device memory, thus avoiding the transmission risks introduced by directly dereferencing the CPU's host pointer.

[0047] Figure 3 This is a flowchart illustrating a tensor buffer transfer method according to an exemplary embodiment of this application. This method can be used in a computer device. Figure 3 As shown, the method includes the following steps.

[0048] Step 302: Receive the cache transfer request.

[0049] A cache transfer request is used to request the transfer of a tensor cache, such as requesting the transfer of a tensor cache from the CPU's host memory to the device memory of an accelerator, or requesting the transfer of a tensor cache from the device memory of an accelerator to the CPU's host memory. The CPU's host memory can be understood as the memory located on the CPU, and the accelerator's device memory can be understood as the memory located on the accelerator.

[0050] A computer device runs an LLM-based inference service, and an acceleration device is used to accelerate the LLM inference process. Exemplarily, the acceleration device includes at least one of a GPU, an NPU, and other devices besides GPUs and NPUs. A tensor cache is used to cache at least one of K tensors and V tensors generated during the LLM inference process. A K tensor can be understood as a feature label for a lexical unit, used to store the semantic features of the lexical unit. The K tensor is used to calculate attention weights to help the LLM filter out other lexical units that need attention for the current lexical unit. A V tensor can be understood as the actual information carrier of a lexical unit, used to store the semantic content of the lexical unit. The V tensor is used to perform a weighted sum based on the attention weights to obtain the output of the LLM's attention layer. In some embodiments, the tensor cache in this application includes a KV cache.

[0051] In some embodiments, the cache transfer request is triggered by the inference framework corresponding to the LLM. The inference framework can be understood as a software system used to deploy and run the LLM, responsible for loading the trained LLM onto the hardware and completing the inference task in a low-latency, high-throughput manner. Inference frameworks typically integrate technologies such as key-value cache management, quantization compression, and batch inference scheduling to significantly improve inference speed and reduce memory consumption while ensuring generation quality.

[0052] Step 304: Transfer the tensor cache from the CPU's host memory to the device memory of the acceleration device via a relay.

[0053] When a cache transfer request indicates that the tensor cache should be transferred from the CPU's host memory to the device memory of the accelerator, the computer device will transfer the tensor cache from the CPU's host memory to the accelerator's device memory via a relay method, which can also be called a staged transfer.

[0054] Transferring a tensor cache from the CPU's host memory to the device memory of the accelerator via a relay method can be understood as transferring the tensor cache from the CPU's host memory to the device memory of the accelerator through multiple steps, or transferring the tensor cache from the CPU's host memory to the device memory of the accelerator by transferring the tensor cache between at least three objects, or transferring the tensor cache from the CPU's host memory to the device memory of the accelerator by accessing the CPU host pointer in a way other than direct transfer of the tensor cache.

[0055] In some embodiments, the relay method is other than directly transferring the tensor cache between the CPU's host memory and the acceleration device's device memory. For example, it could be a relay within the CPU to transfer the tensor cache from the CPU's host memory to the acceleration device's device memory, or a relay within the acceleration device to transfer the tensor cache from the CPU's host memory to the acceleration device's device memory. In some embodiments, the relay method is a cache other than the device cache used to store the tensor cache within the acceleration device's device memory.

[0056] Optionally, the computer device obtains a tensor cache from the CPU's host memory, copies the obtained tensor cache to the temporary cache of the acceleration device, and then writes the tensor cache from the temporary cache into the device cache in device memory, which is used to store the tensor cache. The temporary cache of the acceleration device can be understood as a cache used to bridge the CPU's host memory and the acceleration device's device memory, or as a cache used to relay data transferred between the CPU's host memory and the acceleration device's device memory. The device cache in device memory is the area in device memory used to store the tensor cache. The device cache in device memory and the temporary cache in device memory are different caches in device memory, and can be understood as different areas in device memory.

[0057] Alternatively, the computer device retrieves the tensor cache from the CPU's host memory and writes it to a pre-created intermediate memory region within the CPU. The tensor cache in this intermediate memory region is then transferred to the device memory of the acceleration device. The memory region in the CPU's host memory used to store the tensor cache is a different region from the pre-created intermediate memory region within the CPU, and it differs from CPU page-locked host memory. The intermediate memory region acts as an efficient data bridge between host memory and device memory, handling large volumes of data like tensor caches and supporting the subsequent transfer of tensor cache data to device memory. The intermediate memory region can be understood as a dedicated physical memory region pre-created and allocated during CPU startup or initialization, specifically for data transfer between the CPU and the acceleration device.

[0058] Step 306: Transfer the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay.

[0059] When a cache transfer request indicates that the tensor cache should be transferred from the device memory of the accelerator to the host memory of the CPU, the computer device will transfer the tensor cache from the device memory of the accelerator to the host memory of the CPU through a relay method, which can also be called a staged transfer.

[0060] Transferring a tensor cache from the device memory of the accelerator to the host memory of the CPU via a relay method can be understood as transferring the tensor cache from the device memory of the accelerator to the host memory of the CPU through multiple steps, or transferring the tensor cache from the device memory of the accelerator to the host memory of the CPU by transferring the tensor cache between at least three objects, or transferring the tensor cache from the device memory of the accelerator to the host memory of the CPU by accessing the CPU host pointer in a way other than direct transfer of the tensor cache.

[0061] In some embodiments, the relay method is other than directly transferring the tensor cache between the CPU's host memory and the acceleration device's device memory. For example, it involves transferring the tensor cache from the acceleration device's device memory to the CPU's host memory via an internal CPU relay, or vice versa. In some embodiments, the relay method uses a cache other than the device cache used to store the tensor cache within the acceleration device's device memory.

[0062] Optionally, the computer device obtains the tensor cache through the device cache in device memory used to store tensor caches, copies the obtained tensor cache to the scratch cache of the acceleration device, and then writes the tensor cache from the scratch cache to the CPU's host memory. The scratch cache of the acceleration device can be understood as a cache used to bridge the CPU's host memory and the acceleration device's device memory, or a cache used to relay data transferred between the CPU's host memory and the acceleration device's device memory. The device cache in device memory is the area in device memory used to store tensor caches. The device cache in device memory and the scratch cache in device memory are different caches in device memory, and can be understood as different areas in device memory.

[0063] Alternatively, the computer device can acquire a tensor cache via the device's accelerated memory and transfer the acquired tensor cache to a pre-created intermediate memory region in the CPU. The tensor cache in the intermediate memory region is then read into the CPU's host memory. The memory region in the CPU's host memory used to store the tensor cache is a different memory region from the pre-created intermediate memory region in the CPU, and the pre-created intermediate memory region in the CPU is different from CPU page-locked host memory.

[0064] In summary, the method provided in this embodiment transfers the tensor cache from the CPU's host memory to the accelerator's device memory, or vice versa, via a relay method. This enables the transfer of the tensor cache between the CPU's host memory and the accelerator's device memory through a means other than accessing the CPU host pointer, thereby avoiding the transmission risks introduced by directly dereferencing the CPU host pointer.

[0065] Figure 4 This is a flowchart illustrating another method for transferring tensor buffers provided in an exemplary embodiment of this application. This method can be used in computer devices. Figure 4 As shown, the method includes the following steps.

[0066] Step 402: Receive the cache transfer request.

[0067] A cache transfer request is used to request the transfer of a tensor cache from the CPU's host memory to the device memory of the acceleration device. The CPU's host memory can be understood as the memory located on the CPU, and the acceleration device's device memory can be understood as the memory located on the acceleration device. The computer device runs an LLM-based inference service, and the acceleration device is used to accelerate the LLM inference process. The tensor cache is used to cache at least one of the K tensors and V tensors generated during the LLM inference process. In some embodiments, the tensor cache in this application includes a KV cache.

[0068] Optionally, the host memory of the CPU in this embodiment includes CPU pinned host memory. CPU pinned host memory is CPU-side memory space that is physically marked as non-swappable by the operating system in a heterogeneous computing system. Unlike regular CPU memory, it is not swapped into virtual memory. CPU pinned host memory is used to improve the data transfer efficiency between the CPU host and heterogeneous acceleration devices such as GPUs and NPUs.

[0069] By limiting the CPU host memory to page-locked memory, the heterogeneous transmission acceleration characteristics of CPU page-locked memory can be fully utilized, avoiding the transmission jitter caused by ordinary pageable host memory, and improving the stability of tensor cache transmission and bandwidth utilization between the CPU and acceleration devices.

[0070] In some embodiments, the cache transfer request is triggered by the inference framework corresponding to the LLM. The inference framework can be understood as a software system used to deploy and run the LLM, responsible for loading the trained LLM onto the hardware and completing the inference task in a low-latency, high-throughput manner. Inference frameworks typically integrate technologies such as key-value cache management, quantization compression, and batch inference scheduling to significantly improve inference speed and reduce memory consumption while ensuring generation quality.

[0071] Upon receiving a cache transfer request, the computer device transfers the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay method. The transfer process is described in detail below. Optionally, if the cache transfer request satisfies a first condition, the computer device transfers the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay method. The first condition includes at least one of the following: the cache transfer request instructing the transfer of the tensor cache from the CPU's host memory to the acceleration device's device memory; and a transfer switch being enabled. The transfer switch controls whether the tensor cache is transferred using the method provided in this embodiment.

[0072] For example, a cache transfer request includes at least one of the following: information indicating a host-side K-tensor cache (host key cache), information indicating a host-side V-tensor cache (host value cache), information indicating a device-side K-tensor cache (device key cache), information indicating a device-side V-tensor cache (device value cache), a set of host indices, a set of device indices, information indicating the tensor cache layout (cache layout), and information indicating the data type of the tensor cache. The host-side K-tensor cache and host-side V-tensor cache can be understood as the source data carriers for tensor cache transfer, and the device-side K-tensor cache and device-side V-tensor cache can be understood as the target data carriers for tensor cache transfer. The host indices indicate the location of the host cache in host memory used to store the tensor cache, and the device indices indicate the location of the device cache in device memory used to store the tensor cache. The tensor cache layout can include different layouts such as paged layout and layer-first layout, and the tensor cache data type can be understood as the data precision of the tensor cache. Upon receiving a cache transfer request, the system checks whether the host-side K / V tensor cache is a CPU tensor, whether the device-side K / V tensor cache is an accelerated device tensor, and whether the transfer switch is enabled. If all conditions are met, the tensor cache will be transferred from the CPU's host memory to the accelerated device's device memory via a relay method as described below. If none or some of the above conditions are met, the tensor cache will not be transferred from the CPU's host memory to the accelerated device's device memory via a relay method as described below, or the tensor cache will be transferred from the CPU's host memory to the accelerated device's device memory via another method.

[0073] The routing verification mechanism based on the first condition automatically determines whether the cached transmission request meets the admission requirements for phased transmission. Without modifying the publicly exposed backend parameters of the inference framework, the routing decision can be automatically completed within the operator library, keeping the upper-layer inference framework interface completely stable and reducing the invasiveness and adaptation costs of framework integration.

[0074] Step 404: Obtain the tensor cache through the CPU's host memory.

[0075] In the process of transferring tensor cache from the CPU's host memory to the device memory of the accelerator, the computer device first needs to retrieve the tensor cache to be transferred from the host memory where the tensor cache is stored, thereby realizing the transfer of tensor cache to device memory.

[0076] Optionally, the cache transfer request includes a host index set, which indicates the location of the host cache in host memory used to store the tensor cache. When the indices in the host index set are consecutive, for example, sequentially increasing, the computer device retrieves the tensor cache from the host cache in a first manner, such as through a narrowing operation or a slice operation, to obtain a contiguous memory view corresponding to the indices in the host index set, thus obtaining the tensor cache. The narrowing operation involves extracting consecutive tensor fragments starting from a given starting index (i.e., an index in the host index set) to generate the corresponding contiguous memory view. The slice operation directly extracts memory segments corresponding to the index range in the host index set from the tensors stored in host memory to generate a contiguous memory view. A contiguous memory view is a tensor with completely contiguous memory addresses, corresponding to a data segment in the host cache with a corresponding consecutive index in the host index set. When the indices in the host index set are not contiguous, the computer device retrieves the tensor cache from the host cache using a second method. For example, it uses an index_select operation to retrieve the slots corresponding to the indices in the host index set from the host cache to obtain the tensor cache. The first and second methods differ. For instance, the first method may be faster, more efficient, or consume fewer resources than the second method in retrieving the tensor cache. The index_select operation retrieves the corresponding slots from the host cache one by one according to the indices in the host index set, thus aggregating this data into the tensor cache. A slot is a storage unit used to store the K / V tensor corresponding to a single token; it is the basic granularity for KV cache management.

[0077] By replacing the general index selection operation with box selection and slicing operations for continuous host indexing scenarios, the redundant overhead of index-type operations in continuous slot scenarios is eliminated. The index selection operation is only retained as a fallback in non-contiguous indexing scenarios. Under the premise of ensuring compatibility across all scenarios, the index access time of host-side tensor cache is significantly reduced.

[0078] In some embodiments, the tensor cache obtained through the CPU's host memory includes linear tensors. A linear tensor refers to a tensor stored in a linear memory space, or a tensor with completely contiguous memory addresses. By directly outputting a tensor cache in linear tensor format from the CPU's host memory, it is possible to directly interface with the copying process of subsequent temporary caches without performing additional memory layout conversion, reducing intermediate data processing steps, and simplifying the data flow logic of the tensor cache transmission link.

[0079] Step 406: Copy the acquired tensor cache to the temporary cache of the acceleration device.

[0080] The staging buffer of an accelerator can be understood as a buffer used to bridge the host memory of the CPU and the device memory of the accelerator, or a buffer used to transfer data between the host memory of the CPU and the device memory of the accelerator.

[0081] Optionally, if the target staging tensor exists among the staging tensors included in the acceleration device's staging cache, the computer device copies the acquired tensor cache to the target staging tensor. If the target staging tensor does not exist among the staging tensors included in the acceleration device's staging cache, the computer device allocates the target staging tensor in the staging cache and copies the acquired tensor cache to the target staging tensor. The target staging tensor includes staging tensors that match the tensor cache. The computer device can determine whether the staging tensors in the staging cache match the tensor cache based on the relevant information of the tensor cache in the cache transfer request. Matching a staging tensor with the tensor cache can be achieved by the staging tensor shape, data precision (dtype), and corresponding device of the staging tensor matching the tensor cache. A staging tensor is a reusable tensor existing in the staging cache, i.e., the device's memory, used as a temporary relay carrier during the transmission of the tensor cache. The staging cache can be understood as a cache pool used to store staging tensors.

[0082] By using a reusable temporary cache mechanism, the resource allocation of the acceleration device is avoided for each transmission, which can reduce memory allocation overhead and memory fragmentation. While making full use of the device's memory resources, the overall time consumption of tensor cache transfer is reduced, thus adapting to the frequent tensor cache transfer requirements in high-concurrency inference scenarios.

[0083] Step 408: Write the tensor cache in the temporary cache to the device cache in the device memory used to store the tensor cache.

[0084] The device cache in device memory is a region within device memory used to store tensor caches. The device cache in device memory and the temporary cache in device memory are different caches within device memory, and can be understood as different regions within device memory. By copying the acquired tensor cache to the temporary cache of the acceleration device, the tensor cache is transferred from the CPU to the acceleration device. Furthermore, the tensor cache needs to be placed at the location indicated in device memory by the cache transfer request to write the tensor cache into the device memory of the acceleration device.

[0085] Optionally, the cache transfer request includes a set of device indexes, which indicates the location of the device cache in device memory for storing the tensor cache. When the indices in the device index set are contiguous and the device cache uses a layer-first layout, the computer device writes the tensor cache from the temporary cache to the device cache using a third method, such as through a narrowing operation and a copy operation. The layer-first layout prioritizes organizing the tensor cache data according to the dimensions of the LLM model layers. With a layer-first layout, the tensor cache has a more contiguous physical arrangement in device memory, allowing for direct transfer without additional layout transformation. The narrowing operation selects the contiguous memory view corresponding to the indices in the device index set, and the copy operation copies the tensor cache to the contiguous memory view to write the tensor cache to the device cache. When the indices in the device index set are not contiguous, or the device cache is not laid out in a layer-first manner, the computer device writes the tensor cache from the temporary cache to the device cache using a fourth method. For example, this can be achieved by using the first operator of the acceleration device. The third and fourth methods differ. For instance, the third method may write the tensor cache faster than the fourth method, be more efficient, or consume fewer resources. The first operator adapts the layout of the tensor cache to the device cache. For example, it may convert the layout format of the tensor cache to match that of the device cache, such as converting it to a paged layout or other layouts, before writing the converted tensor cache to the device cache. The first operator is a native operator of the acceleration device, which is a dedicated operator developed specifically for acceleration devices. Native operators are fully adapted to the hardware characteristics of acceleration devices, enabling efficient handling of non-contiguous data and format conversion between different cache layouts within device memory. Compared to the general operators provided by the inference framework, native operators can fully unleash the computational and data transfer performance of the acceleration device.

[0086] If the indexes in the device index set are contiguous and the device cache uses a layer-first layout, it can be understood as satisfying the fast path condition. If the indexes in the device index set are not contiguous or the device cache does not use a layer-first layout, it can be understood as not satisfying the fast path condition. The fast path can avoid unnecessary layout format conversions and general operator scheduling overhead, significantly improving data movement efficiency and offering a significant speed improvement compared to the non-optimized path.

[0087] For device indexing scenarios with continuous, layer-priority layouts, a fast path of selection and copying is directly used to replace the general layout conversion operator, eliminating unnecessary format conversion overhead between paginated and linear layouts. By retaining fallback logic for operator adaptation in non-contiguous or special layout scenarios, the system maximizes device-side data movement efficiency while ensuring compatibility with various cache layouts, thereby reducing the time spent in the layout transfer process.

[0088] In some embodiments, during the transfer of the tensor cache from the CPU's host memory to the device memory of the acceleration device, the computer device also collects a first statistical indicator related to the tensor cache transfer process. Optionally, the first statistical indicator includes at least one of CPU indexing timing, Host-to-Device (H2D) copy timing, and layout movement timing. CPU indexing timing reflects the time spent retrieving the tensor cache from the CPU's host memory, H2D copy timing reflects the time spent copying the tensor cache from host memory to device memory, and layout movement timing reflects the time spent adjusting the cache layout on the acceleration device side. Optionally, the computer device collects the first statistical indicator through a strong diagnostic mode or a low-intrusion profiling mode. Strong diagnostic mode performs explicit synchronization at the acceleration device's operational boundary, while low-intrusion profiling mode does not enforce explicit synchronization. Low-intrusion profiling mode has less interference with normal business processes compared to strong diagnostic mode. In strong diagnostic mode, the computer device can also collect synchronization timing, which reflects the time taken to perform the explicit synchronization described above.

[0089] In summary, the method provided in this embodiment transfers the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay mechanism. This allows for the transfer of the tensor cache from the CPU's host memory to the acceleration device's device memory through a method other than accessing the CPU's host pointer, thus avoiding the transmission risks introduced by directly dereferencing the CPU's host pointer. By splitting the transmission process into stages—acquiring the tensor cache, copying it to a temporary cache, and writing it to device memory—a secure bridge channel is established between host memory and device memory. This solves the problem of incompatibility between host memory and the acceleration device's native operator memory types, allowing the acceleration device's native operator library, which originally only supported device-side tensor input, to stably take over tensor cache transmission in host memory scenarios, avoiding runtime errors caused by memory type incompatibility.

[0090] Figure 5This is a flowchart illustrating another tensor buffer transfer method provided in an exemplary embodiment of this application. This method can be used in computer devices. Figure 5 As shown, the method includes the following steps.

[0091] Step 502: Receive the cache transfer request.

[0092] A cache transfer request is used to request the transfer of a tensor cache from the device memory of the acceleration device to the host memory of the CPU. The host memory of the CPU can be understood as the memory located on the CPU, and the device memory of the acceleration device can be understood as the memory located on the acceleration device. The computer device runs an LLM-based inference service, and the acceleration device is used to accelerate the LLM inference process. The tensor cache is used to cache at least one of the K tensors and V tensors generated during the LLM inference process. In some embodiments, the tensor cache in this application includes a KV cache. Optionally, the host memory of the CPU in this application includes CPU page-locked host memory.

[0093] In some embodiments, the cache transfer request is triggered by the inference framework corresponding to the LLM. The inference framework can be understood as a software system used to deploy and run the LLM, responsible for loading the trained LLM onto the hardware and completing the inference task in a low-latency, high-throughput manner. Inference frameworks typically integrate technologies such as key-value cache management, quantization compression, and batch inference scheduling to significantly improve inference speed and reduce memory consumption while ensuring generation quality.

[0094] Upon receiving a cache transfer request, the computer device transfers the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay method. The transfer process can be referred to the relevant content below. Optionally, if the cache transfer request satisfies a second condition, the computer device transfers the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay method. The second condition includes at least one of the following: the cache transfer request instructing the transfer of the tensor cache from the device memory of the acceleration device to the host memory of the CPU, and a transfer switch being enabled. The transfer switch is used to control whether the tensor cache is transferred using the method provided in this application embodiment. A detailed description of the cache transfer request can be found in the relevant content above; it will not be repeated here.

[0095] Through the routing verification mechanism based on the second condition, legitimate requests in the reverse offloading scenario are automatically identified. The routing judgment is completed independently within the operator library of the acceleration device. It can automatically take over the reverse transmission process without the need for additional special configuration in the upper-layer framework, thus maintaining the compatibility and stability of the upper-layer inference framework interface.

[0096] Step 504: Obtain the tensor cache through the device cache used to store tensor caches in the device memory.

[0097] The device cache in device memory is a region in device memory used to store tensor caches. The device cache in device memory and the temporary cache in device memory are different caches in device memory, and can be understood as different regions in device memory.

[0098] Optionally, the cache transfer request includes a set of device indexes, which indicates the location of the device cache in the device memory used to store the tensor cache. If the indices in the device index set are contiguous and the device cache has a layer-first layout, the computer device obtains the tensor cache using a fifth method, such as directly retrieving the slice corresponding to the index in the device index set from the device cache. If the indices in the device index set are not contiguous, or the device cache does not have a layer-first layout, the computer device obtains the tensor cache using a sixth method, such as retrieving the tensor cache from the device cache using a second operator that accelerates the device. The fifth and sixth methods differ. For example, the fifth method may be faster than the sixth method, more efficient than the sixth method, or consume fewer resources than the sixth method. The second operator is used to adjust the layout of the data in the device cache, for example, to convert the layout format of the data in the device cache from paged to linear, in order to obtain the tensor cache. The second operator is a native operator for accelerated devices. Native operators are specialized operators developed for accelerated devices, and the second operator can be called the paged-to-linear operator. It should be noted that the acquisition of tensor caches can be found in the corresponding content above.

[0099] By directly slicing the device index for scenarios with priority to continuous layer layout, redundant layout conversion processes are skipped. The second operator is used to adjust the layout only in scenarios with non-continuous index or non-target layout. While being compatible with various device cache layouts, the overhead of the device-side data extraction process is reduced, and the transmission efficiency of the tensor cache offloading link is improved.

[0100] In some embodiments, the tensor cache obtained by accelerating the device's memory includes linear tensors. A linear tensor is a tensor stored in a linear memory space, or a tensor with completely contiguous memory addresses. By directly outputting a tensor cache in linear tensor format from the device memory, it can be directly written to the temporary cache without additional data format conversion, simplifying the data flow logic of the tensor cache transmission link and reducing unnecessary computational overhead.

[0101] Step 506: Copy the acquired tensor cache to the temporary cache of the acceleration device.

[0102] The staging buffer of an accelerator can be understood as a buffer used to bridge the host memory of the CPU and the device memory of the accelerator, or a buffer used to transfer data between the host memory of the CPU and the device memory of the accelerator.

[0103] Optionally, if the target staging tensor exists in the staging tensors included in the acceleration device's staging cache, the computer device copies the acquired tensor cache to the target staging tensor. If the target staging tensor does not exist in the staging tensors included in the acceleration device's staging cache, the computer device allocates the target staging tensor in the staging cache and copies the acquired tensor cache to the target staging tensor. The target staging tensor includes staging tensors that match the tensor cache. The computer device can determine whether the staging tensors in the staging cache match the tensor cache based on the relevant information in the staging cache in the cache transfer request. It should be noted that the copying from the tensor cache to the staging cache can be referred to the corresponding content above.

[0104] By reusing the temporary cache in the reverse transmission link from device to host, the repeated allocation of the temporary cache in the reverse scenario is avoided, the memory management overhead of the tensor cache transmission link is reduced, the resource utilization of the bidirectional transmission process is kept consistent, and the scheduling stability of device-side resources in high-concurrency scenarios is improved.

[0105] Step 508: Write the tensor cache in the temporary cache to the CPU's host memory.

[0106] Optionally, the cache transfer request includes a host index set, which indicates the location of the host cache in host memory used to store the tensor cache. The host cache in host memory is the region in host memory used to store the tensor cache. The computer device first copies the tensor cache in the temporary cache to the CPU's pinned memory. If the indices in the host index set are contiguous, the computer device writes the tensor cache in the CPU's pinned memory back to the host cache using a seventh method, such as using a narrowing operation and a copy operation. The narrowing operation is used to select a contiguous memory view corresponding to an index in the host index set, and the copy operation is used to copy the tensor cache to a contiguous memory view. If the indices in the host index set are not contiguous, the computer device writes the tensor cache in the CPU's pinned memory back to the host cache using an eighth method, such as using an index copy operation to write the tensor cache in the CPU's pinned memory back to the slots corresponding to the indices in the host index set. The seventh and eighth methods are different. For example, method 7 may be faster than method 8 in writing back to the tensor cache, more efficient in writing back to the tensor cache, or consume fewer resources than method 8 in writing back to the tensor cache. It should be noted that the corresponding content in the previous text can be referred to for instructions on writing to the tensor cache.

[0107] By first copying the tensor cache on the device side to CPU page locked memory, and then adapting different write-back logic based on index continuity: in the case of continuous indexes, the fast path is copied by box selection to reduce write-back overhead, and in the case of non-continuous indexes, index copying is used as a fallback to ensure correctness, making full use of the high transmission efficiency advantage of CPU page locked memory, while taking into account the write-back performance and compatibility on the host side.

[0108] In some embodiments, during the transfer of the tensor cache from the device memory of the acceleration device to the host memory of the CPU, the computer device also collects a second statistical indicator related to the transfer process of the tensor cache. Optionally, the second statistical indicator includes at least one of layout operation timing, device-to-host (D2H) copy timing, and CPU index copy timing. The layout operation timing reflects the process time of the layout adjustment operation on the acceleration device side, the D2H copy timing reflects the time spent copying the tensor cache from device memory to host memory, and the CPU index copy timing reflects the operation time of writing the tensor cache copied from D2H back to the host cache. Optionally, the computer device collects the second statistical indicator through a strong diagnostic mode or a low-intrusion profiling mode. The strong diagnostic mode performs explicit synchronization at the operation boundary of the acceleration device, while the low-intrusion profiling mode does not enforce explicit synchronization. The low-intrusion profiling mode has less interference with normal business processes compared to the strong diagnostic mode. In strong diagnostic mode, the computer device can also collect synchronization time, which reflects the time taken to perform the explicit synchronization described above.

[0109] In summary, the method provided in this embodiment transfers the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay method. This achieves the transfer of the tensor cache from the device memory of the acceleration device to the host memory of the CPU through a means other than accessing the CPU host pointer, thereby avoiding the transmission risks introduced by directly dereferencing the CPU host pointer. By breaking down the tensor cache offloading process from device to host into phased steps of fetching data from device memory, copying to a temporary cache, and writing to host memory, and without relying on the acceleration device kernel to directly derefer to the CPU host pointer, the method fundamentally avoids the security risks such as address space, data consistency, and abnormal access caused by the device directly accessing host memory, significantly improving the universality and verifiability of data transmission in heterogeneous systems.

[0110] This application provides a CPU host staged KV cache transfer method and system for LLM inference systems. When the inference framework's host KV cache is located in the CPU's page-locked memory and the device KV cache is located in the acceleration device's device memory, the system can securely and efficiently take over data transfer between the host cache and the device cache through a Route C (representing the KV cache transfer method in this application) routing mechanism. This method ensures memory type correctness while utilizing CPU contiguous slot fast paths, reusable staging buffers of the acceleration device, layer-first layout fast paths of the acceleration device, and low-intrusion profiling mechanisms to reduce transmission overhead and improve diagnostic accuracy.

[0111] For example, Figure 6 This is a schematic diagram of a system architecture provided in an exemplary embodiment of this application. For example... Figure 6 As shown, system 600 includes a framework integration layer 601, a routing decision layer 602, a CPU host slot access layer 603, a device temporary cache management layer 604, a device layout conversion layer 605, a device native operator layer 606, and a diagnosis and analysis layer 607.

[0112] Framework Integration Layer 601: Used to interface with the LLM inference framework, receiving KV cache transfer requests, cache tensors, slot indexes, layout metadata, and backend configurations from the framework side. Integration methods can be found in [reference needed]. Figure 7 As shown. Figure 7 As shown, LLM service requests pass through the inference framework, cache management, and system integration adapter in sequence before entering the routing decision layer. Then, through CPU host phased transmission and device native operators, data is transmitted between the CPU page-locked host cache, device temporary cache, and device KV cache.

[0113] Routing decision layer 602: Used to determine whether the current request meets the conditions of route C, including whether the host cache is a CPU tensor, whether the device cache is a device tensor, whether the cache layout is supported, and whether the environment switch is enabled.

[0114] CPU host slot access layer 603: Used to select or write back the corresponding term slot from the CPU page-locked host cache based on the host index set. For contiguous slot ranges, a slice or box selection copy operation is used; for non-contiguous slots, indexed operation fallback is retained.

[0115] Device temporary cache management layer 604: Used to allocate or reuse temporary tensors on device memory as a data bridging buffer between the CPU host cache and the device cache.

[0116] Device layout transformation layer 605: Used to write linear key-value tensors from temporary tensors to the device key-value cache, or to select the corresponding slot from the device key-value cache and write back the temporary tensor. For continuous layer-priority slots, a layer-priority slicing fast path is used; for non-contiguous or special layouts, a paging kernel fallback is used.

[0117] Device native operator layer 606: includes device native operators such as KV paging to linear, linear to paging, paging block copy, and FP8 KV loading.

[0118] Diagnostics and Analysis Layer 607: Used to output routing decisions, replication time, layout time, and synchronization time. It includes at least a strong diagnostic mode and a low-intrusion analysis mode.

[0119] For example, Figure 8 This is a routing C data flow graph provided in an exemplary embodiment of this application. (See reference...) Figure 6 and Figure 8 When it is necessary to load key-value data from the CPU page-locked cache into the device cache, perform the following steps.

[0120] Step 1: The framework integration layer receives the host-to-device key-value transfer request. This request includes host key cache, host value cache, device key cache, device value cache, host indices, device indices, cache layout, and data type.

[0121] Step 2: The routing decision layer determines whether the host key / value cache is a CPU tensor, whether the device key / value cache is a device tensor, and whether route C is enabled. If the conditions are met, route C is selected; otherwise, it returns to not take over or proceeds to another valid path.

[0122] Step 3: The CPU host slot access layer converts host indices into CPU long indices and determines whether the indices form a continuously increasing range. If they are continuous, it obtains a continuous view from the host cache using selection or slicing; if they are not continuous, it obtains the corresponding slot using index selection. The CPU host slot access layer can select slicing, gathering, scattering, batch copying, or vectorized copying based on the index pattern.

[0123] Step 4: The device staging cache management layer obtains reusable device staging tensors based on tensor shape, data precision, and device. If a matching staging tensor already exists in the cache pool, it is reused; otherwise, a new staging tensor is allocated on the accelerated device. The device staging cache management layer can establish cache pools with different granularities by layer, data precision, shape, device, or request group. When secure direct access to CPU page-locked host pointers is supported, staging replication can be replaced with direct CPU page-locked host kernel, while still retaining routing C decision and fallback mechanisms.

[0124] Step 5: Copy the CPU linear KV tensor obtained in Step 3 to the device temporary storage tensor obtained in Step 4 to obtain the device linear KV tensor.

[0125] Step 6: The device layout transformation layer determines whether device indices are contiguous. If they are contiguous and the device cache is a layer-first layout, it uses bounding selection copying to write the device linear KV tensor to the device cache; if the fast path condition is not met, it calls the device's native KV operator to write the linear tensor to a paging layout or other target layout. The device layout transformation layer can select a layer-first slice, a paged kernel, a multi-layer kernel, or a fused kernel based on the layout type.

[0126] Step 7: The diagnostic and analysis layer records CPU indexing timing, H2D copy timing, layout movement timing, and optional sync timing according to the configuration.

[0127] Step 8: Return the result of successful route execution, and the framework continues the subsequent attention calculation or generation process.

[0128] refer to Figure 6 and Figure 8When it is necessary to load key-value data from the CPU page-locked cache into the device cache, perform the following steps.

[0129] Step 1: The framework integration layer receives the device-to-host key-value transfer request. This request includes device key / value cache, host key / value cache, device indices, host indices, layout, and dtype.

[0130] Step 2: The routing decision layer confirms that the host cache is CPU page locked memory and the device cache is device memory, and selects route C.

[0131] Step 3: The device layout conversion layer determines whether device indices are continuous. If they are continuous and the layout is layer-first, the corresponding slot's key-value slice is directly obtained from the device cache; if they are not continuous or the layout does not meet the requirements, the native paging to linear operator is called to obtain the device's linear key-value tensor.

[0132] Step 4: The device temporary cache management layer copies the device linear key-value tensor to the CPU page locked memory, thus obtaining the CPU linear key-value tensor.

[0133] Step 5: The CPU host slot access layer determines whether host indices are contiguous. If they are contiguous, the CPU linear tensor is written back to the host cache using a box selection copy; if they are not contiguous, the index copy is used to write them back to the host cache.

[0134] Step 6: Diagnostic and analysis layer records layout operation, D2H copy, CPU index copy, and optional sync timing.

[0135] Step 7: Return the result of successful execution of route C.

[0136] For example, Figure 9 This is a schematic diagram of a diagnostic mode provided in an exemplary embodiment of this application. Figure 9 As shown in the embodiments of this application, at least two types of diagnostic modes are also provided.

[0137] Strong diagnostic mode: Enables route logs and performs explicit synchronization at critical device operation boundaries to obtain detailed logs including sync timing.

[0138] Low-intrusion analysis mode: Enables route profile logs but does not force explicit synchronization, used to observe latency and throughput closer to production mode. Low-intrusion analysis mode can output to logs, in-memory ring buffers, metrics collectors, or tracing systems.

[0139] By distinguishing between metrics such as layout_op_ms, layout_sync_ms, d2h_copy_ms, h2d_copy_ms, and h2d_sync_ms, it is possible to determine whether outliers originate from actual copy / layout stalls or are primarily caused by measurement artifacts resulting from explicit synchronization.

[0140] For example, Figure 10 This is a schematic diagram illustrating the transmission process of a KV buffer provided in an exemplary embodiment of this application. For example... Figure 10 As shown, taking the example of enabling hierarchical caching in the inference framework and using hicache-io-backendkernel, the host KV cache is allocated in CPU page-locked memory by the inference framework, and the device KV cache is located in the device memory of the acceleration device.

[0141] When a request needs to load a key-value pair from the host cache to the device cache: Step S1: The inference framework calls the framework integration layer, passing in host cache, device cache, hostindices, and device indices; Step S2: The routing decision layer detects that the host cache is a CPU tensor and the device cache is a device tensor, and selects route C; Step S3: The CPU host slot access layer determines that host indices are a continuous range and uses slicing to obtain the KV linear tensor; Step S4: The temporary cache management layer reuses temporary tensors; Step S5: The CPU linear tensor is copied to a temporary tensor; Step S6: The layout transformation layer determines whether device indices are continuous and writes them to the device cache using layer-first slicing; Step S7: The system records diagnostic / analysis information during transmission; Step S8: The system returns success, and subsequent attention calculations will use the restored device KV cache.

[0142] In another embodiment, when the device-side KV cache needs to be offloaded to the CPU host cache: Step A1: The inference framework is fed in device cache, host cache, device indices, and hostindices; Step A2: The routing decision layer selects route C; Step A3: The layout transformation layer selects contiguous layer slices from the device cache with priority; Step A4: The selected KV tensor is copied from the device to the CPU and written to the CPU linear tensor; Step A5: The CPU host slot access layer determines that host indices are continuous, and uses a selection box to copy and write back to the CPU page to lock the host cache; Step A6: Complete the relocation.

[0143] In one embodiment, the system first enables a strong diagnostic mode, observing a high total maximum timing for route C; subsequently, a low-intrusion analysis mode is enabled, without explicit synchronization. Comparison reveals that the maximum maximum timing in the strong diagnostic mode is significantly higher than in the low-intrusion analysis mode, while p50 remains largely consistent. This indicates that the high maximum maximum timing primarily stems from diagnostic synchronization artifacts, rather than actual transfer stalls on the production path.

[0144] The embodiments of this application include at least the following beneficial effects.

[0145] To resolve the mismatch between CPU host cache and device operator memory type: By using device temporary tensors as a bridging layer, the device native operator library can safely take over CPU page locking host cache scenarios.

[0146] Reduce CPU host indexing overhead: Use slicing or box selection copying for contiguous slot ranges to reduce the additional overhead of indexing operations.

[0147] Reduce device layout conversion overhead: Use direct slice fast path for continuous layer priority device slots to avoid unnecessary layer priority and pagination layout conversions.

[0148] To maintain the stability of the inference framework user interface: no new public backend parameters are needed; the existing kernel backend of the framework will continue to be used, and routing decisions will be made internally by the system.

[0149] Improve diagnostic accuracy: By separating copy / layout time from synchronization time, avoid misjudging measurement artifacts introduced by explicit synchronization as real transmission pauses.

[0150] Improved production stability: It can still run stably in production mode with diagnostics and analysis turned off, and there are no rollback logs.

[0151] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0152] It should be noted that the order of the method steps provided in the embodiments of this application can be appropriately adjusted, and the steps can also be added or removed as appropriate. Any method variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0153] Figure 11 This is a schematic diagram of the structure of a tensor buffer transmission device provided in an exemplary embodiment of this application. Figure 11 As shown, the device includes the following modules.

[0154] The receiving module 1101 is configured to receive a cache transmission request, wherein the cache transmission request is used to request the transmission of the tensor cache, wherein the tensor cache is used to cache at least one of the K tensor and V tensor generated during the LLM inference process; Transmission module 1102 is used to transfer the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay method; or, The tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay method; The acceleration device is used to accelerate the inference process of the LLM.

[0155] In an optional design, the transmission module 1102 is used for: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache into the device cache in the device memory used to store the tensor cache.

[0156] In an optional design, the transmission module 1102 is used for: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

[0157] In an optional design, the cache transfer request includes a set of device indexes, which indicates the location of the device cache in the device memory used to store the tensor cache; the transfer module 1102 is configured to: When the indices in the device index set are consecutive and the device cache is in a layer-first layout, the tensor cache in the temporary cache is written to the device cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the device index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache in the temporary cache is written into the device cache by the first operator of the acceleration device. The first operator is used to adapt the layout of the tensor cache to the layout of the device cache.

[0158] In an optional design, the cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; the transfer module 1102 is configured to: When the indices in the host index set are consecutive, the contiguous memory view corresponding to the indices in the host index set is obtained from the host cache through a selection operation or a slicing operation to obtain the tensor cache; When the indices in the host index set are not contiguous, the slots corresponding to the indices in the host index set are obtained from the host cache through an index selection operation to obtain the tensor cache.

[0159] In an optional design, the tensor cache accessed via the CPU's host memory includes linear tensors.

[0160] In an optional design, the transmission module 1102 is used for: The tensor cache is obtained through the device cache in the device memory used to store the tensor cache; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache to the host memory of the CPU.

[0161] In an optional design, the cache transfer request includes a set of device indexes, which indicates the location of the device cache in the device memory used to store the tensor cache; the transfer module 1102 is configured to: When the indices in the device index set are consecutive and the device cache is a layer-first layout, the slice corresponding to the index in the device index set is directly obtained from the device cache to obtain the tensor cache; If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache is obtained from the device cache by the second operator of the acceleration device, the second operator being used to adjust the layout of the data in the device cache.

[0162] In an alternative design, the tensor cache obtained through the device memory of the acceleration device includes linear tensors.

[0163] In an optional design, the transmission module 1102 is used for: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

[0164] In an optional design, the cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; the transfer module 1102 is configured to: Copy the tensor cache in the temporary cache to the page-locked memory of the CPU; When the indexes in the host index set are consecutive, the tensor cache in the page-locked memory of the CPU is written back to the host cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the host index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indexes in the host index set are not contiguous, the tensor cache in the page-locked memory of the CPU is written back to the slots corresponding to the respective indexes in the host index set through an index copy operation, so as to write the tensor cache back to the host cache.

[0165] In an optional design, the transmission module 1102 is used for: If the cache transfer request meets the first condition, the tensor cache is transferred from the host memory of the CPU to the device memory of the acceleration device via a relay. The first condition includes at least one of the following: the cache transfer request instructing the CPU's host memory to transfer the tensor cache to the acceleration device's device memory, and the transfer switch being enabled.

[0166] In an optional design, the transmission module 1102 is used for: If the cache transfer request meets the second condition, the tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay. The second condition includes at least one of the following: the cache transfer request instructing the tensor cache to be transferred from the device memory of the acceleration device to the host memory of the CPU; and the transfer switch being enabled.

[0167] In an optional design, the CPU's host memory includes CPU page-locked host memory.

[0168] In an optional design, the transmission module 1102 is used for: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is written into a pre-created intermediate memory region in the CPU; The tensor cache in the transit memory region is transferred to the device memory of the acceleration device.

[0169] In an optional design, the transmission module 1102 is used for: The tensor cache is obtained through the device memory of the acceleration device; The acquired tensor cache is transferred to a pre-created intermediate memory region in the CPU; The tensor cache in the intermediate memory region is read into the host memory of the CPU.

[0170] It should be noted that the tensor buffer transmission device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the tensor buffer transmission device and the tensor buffer transmission method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0171] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the tensor cache transfer method provided in the above-described method embodiments.

[0172] For example, Figure 12 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application, which can be implemented as a server.

[0173] The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including random access memory (RAM) 1202 and read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. The computer device 1200 also includes an input / output system 1206 (I / O system) to facilitate information transfer between various components within the computer device, and a mass storage device 1207 for storing the operating system 1213, application programs 1214, and other program modules 1215.

[0174] The input / output system 1206 includes a display 1208 for displaying information and an input device 1209 for user input, such as a mouse or keyboard. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The input / output system 1206 may also include the input / output controller 1210 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0175] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable storage media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include computer-readable storage media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0176] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1204 and mass storage device 1207 described above can be collectively referred to as memory.

[0177] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1201. The one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 1201 executes the one or more programs to implement the methods provided by the above method embodiments.

[0178] According to various embodiments of this application, the computer device 1200 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1200 can be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer device systems (not shown).

[0179] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0180] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the tensor buffer transmission method provided in the above-described method embodiments is implemented.

[0181] This application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the tensor buffer transfer method provided in the above-described method embodiments.

[0182] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0183] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for transferring tensor buffers, characterized in that, The method includes: Receive a cache transfer request, the cache transfer request being used to request the transfer of the tensor cache, the tensor cache being used to cache at least one of the key K tensor and value V tensor generated during the Large Language Model (LLM) inference process; The tensor cache is transferred from the CPU's host memory to the acceleration device's device memory via a relay; or, The tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay method; The acceleration device is used to accelerate the inference process of the LLM.

2. The method according to claim 1, characterized in that, The step of transferring the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay includes: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache into the device cache in the device memory used to store the tensor cache.

3. The method according to claim 2, characterized in that, The step of copying the acquired tensor cache to the temporary cache of the acceleration device includes: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

4. The method according to claim 2, characterized in that, The cache transfer request includes a set of device indexes, which indicates the location of the device cache in the device memory used to store the tensor cache. The step of writing the tensor cache in the temporary cache into the device cache used for storing the tensor cache in the device memory includes: When the indices in the device index set are consecutive and the device cache is in a layer-first layout, the tensor cache in the temporary cache is written to the device cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the device index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache in the temporary cache is written into the device cache by the first operator of the acceleration device. The first operator is used to adapt the layout of the tensor cache to the layout of the device cache.

5. The method according to claim 2, characterized in that, The cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; The step of obtaining the tensor cache through the CPU's host memory includes: When the indices in the host index set are consecutive, the contiguous memory view corresponding to the indices in the host index set is obtained from the host cache through a selection operation or a slicing operation to obtain the tensor cache; When the indices in the host index set are not contiguous, the slots corresponding to the indices in the host index set are obtained from the host cache through an index selection operation to obtain the tensor cache.

6. The method according to claim 1, characterized in that, The step of transferring the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay includes: The tensor cache is obtained through the device cache in the device memory used to store the tensor cache; The acquired tensor cache is copied to the temporary cache of the acceleration device; Write the tensor cache in the temporary cache to the host memory of the CPU.

7. The method according to claim 6, characterized in that, The cache transfer request includes a set of device indexes, which indicates the location of the device cache in the device memory used to store the tensor cache. The step of obtaining the tensor cache through the device cache used to store the tensor cache in the device memory includes: When the indices in the device index set are consecutive and the device cache is a layer-first layout, the slice corresponding to the index in the device index set is directly obtained from the device cache to obtain the tensor cache; If the indices in the device index set are not contiguous, or if the device cache is not laid out according to the layer priority, the tensor cache is obtained from the device cache by the second operator of the acceleration device, the second operator being used to adjust the layout of the data in the device cache.

8. The method according to claim 6, characterized in that, The step of copying the acquired tensor cache to the temporary cache of the acceleration device includes: If a target temporary tensor exists among the temporary tensors included in the temporary cache of the acceleration device, the acquired tensor cache is copied to the target temporary tensor. If the target temporary tensor is not present in the temporary tensors included in the temporary cache of the acceleration device, the target temporary tensor is allocated in the temporary cache, and the obtained tensor cache is copied to the target temporary tensor. The target temporary tensor includes temporary tensors that match the tensor cache.

9. The method according to claim 6, characterized in that, The cache transfer request includes a host index set, which indicates the location of the host cache in the host memory used to store the tensor cache; The step of writing the tensor cache in the temporary cache into the host memory of the CPU includes: Copy the tensor cache in the temporary cache to the page-locked memory of the CPU; When the indexes in the host index set are consecutive, the tensor cache in the page-locked memory of the CPU is written back to the host cache through a selection operation and a copy operation. The selection operation is used to select the consecutive memory view corresponding to the index in the host index set, and the copy operation is used to copy the tensor cache to the consecutive memory view. If the indexes in the host index set are not contiguous, the tensor cache in the page-locked memory of the CPU is written back to the slots corresponding to the respective indexes in the host index set through an index copy operation, so as to write the tensor cache back to the host cache.

10. The method according to any one of claims 1 to 9, characterized in that, The step of transferring the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay includes: If the cache transfer request meets the first condition, the tensor cache is transferred from the host memory of the CPU to the device memory of the acceleration device via a relay. The first condition includes at least one of the following: the cache transfer request instructing the CPU's host memory to transfer the tensor cache to the acceleration device's device memory, and the transfer switch being enabled.

11. The method according to any one of claims 1 to 9, characterized in that, The step of transferring the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay includes: If the cache transfer request meets the second condition, the tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay. The second condition includes at least one of the following: the cache transfer request instructing the tensor cache to be transferred from the device memory of the acceleration device to the host memory of the CPU; and the transfer switch being enabled.

12. The method according to claim 1, characterized in that, The step of transferring the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay includes: The tensor cache is obtained through the host memory of the CPU; The acquired tensor cache is written into a pre-created intermediate memory region in the CPU; The tensor cache in the transit memory region is transferred to the device memory of the acceleration device.

13. The method according to claim 1, characterized in that, The step of transferring the tensor cache from the device memory of the acceleration device to the host memory of the CPU via a relay includes: The tensor cache is obtained through the device memory of the acceleration device; The acquired tensor cache is transferred to a pre-created intermediate memory region in the CPU; The tensor cache in the intermediate memory region is read into the host memory of the CPU.

14. A transmission device for a tensor buffer, characterized in that, The device includes: A receiving module is configured to receive a cache transmission request, wherein the cache transmission request is used to request the transmission of the tensor cache, wherein the tensor cache is used to cache at least one of the K tensor and V tensor generated during the LLM inference process; A transmission module is used to transfer the tensor cache from the CPU's host memory to the acceleration device's device memory via a relay method; or, The tensor cache is transferred from the device memory of the acceleration device to the host memory of the CPU via a relay method; The acceleration device is used to accelerate the inference process of the LLM.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the tensor buffer transfer method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one program segment, which is loaded and executed by a processor to implement the tensor cache transfer method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the tensor buffer transfer method as described in any one of claims 1 to 13.

Images