Methods and apparatus for processing an inference

Abstract

A method of processing an inference of a large language model, LLM, is disclosed together with corresponding apparatus for the same. The method comprises prefilling to generate key value, KV, data; caching multiple copies of the KV data, including at least one copy at a first processing node and at least a further copy at a second processing node; and depending on the workload and / or memory utilisation of the first and second processing nodes, decoding the cached KV data at either the first processing node or the second processing node.

Description

[0001] METHODS AND APPARATUS FOR PROCESSING AN INFERENCE

[0002] FIELD OF THE INVENTION

[0003] This invention relates to methods and apparatus for processing an inference.

[0004] BACKGROUND TO THE INVENTION

[0005] In a large language model (LLM), tokens typically define respective permutations of a combination of language characters. A natural language LLM input or query is first converted in a serious of corresponding input tokens. Thereafter, these input tokens are processed to predict the next token(s) - an ‘inference’.

[0006] Such processing typically involves two steps: prefilling and decoding. Prefilling refers to the processing of input tokens to compute intermediate data (keys and values), commonly known as KV data. Decoding refers to the processing of KV data to determine next token(s). For example, the next token(s) may identify the most likely word or phrase to follow a natural language input, which may be applied to text auto-completion, speech recognition, machine translation and the like.

[0007] An LLM is typically characterised by millions of parameters and therefore the computing requirement to process an LLM inference is correspondingly intensive, both in terms of memory and processing power. This is why the processing of a LLM inference is often done in a cloud environment, on a cloud infrastructure comprising many Al accelerator cards.

[0008] An objective of the present invention is to provide alternative and / or improved methods and apparatus for such prefiling and decoding.

[0009] SUMMARY OF INVENTION

[0010] In accordance with the present invention, methods and apparatus are provided as claimed in the accompanying claims.

[0011] In accordance with a first aspect of the present invention, there is provided a method of processing an inference of a large language model, LLM, comprising: prefilling to generate key value, KV, data; caching multiple copies of the KV data, including at least one copy at a first processing node and at least a further copy at a second processing node; and depending on the workload and / or memory utilisation of the first and second processing nodes, decoding the cached KV data at either the first processing node or the second processing node. Copies of the cached KV data may be synchronised.

[0012] One processing node may be solely responsible for prefilling. Where this is the case, the one processing node solely responsible for prefilling may also share decoding with the other processing node(s) to reduce idling when not prefilling.

[0013] A plurality of pairs of processing nodes may be provided, wherein, each pair of processing nodes processes an inference whereby copies of cached KV data are synchronised between processing nodes of each pair of processing nodes. Where this is the case, upon completion of the processing of an inference by one pair of processing nodes, copies of cached KV data may be transferred to another pair of processing nodes for processing a further inference using the transferred cached KV data.

[0014] In accordance with a second aspect of the present invention, there is provided apparatus comprising at least first and second processing nodes, the apparatus being configured to perform a such method.

[0015] In particular, the at least first and second processing nodes of such apparatus may each comprise: a data processing unit, DPU, configured to provide hardware acceleration for decoding in the processing of an inference of a LLM, wherein each DPU comprises a respective cache.

[0016] Where this is the case, a host processor may be provided and configured to control the DPUs of the processing nodes and to determine, depending on the workload and / or memory utilisation of the first and second processing nodes, which processing node is to process KV data from its respective cache.

[0017] Copies of the cached KV data in the respective caches of the processing nodes may be synchronised.

[0018] One processing node may be solely responsible for prefilling. Where this is the case, the one processing node solely responsible for prefilling also shares decoding with the other processing nodes to reduce idling when not prefilling.

[0019] A plurality of pairs of processing nodes may be provided, wherein each pair of processing nodes is configured to process an inference whereby copies of cached KV data are synchronised between processing nodes of each pair of processing nodes. Where this is the case, the apparatus may be further configured to, upon completion of the processing of an inference by one pair of processing nodes, transfer copies of cached KV data to another pair of processing nodes for processing a further inference using the transferred KV data.

[0020] In accordance with a third aspect of the present invention, there is provided a DPU of such apparatus.

[0021] BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 illustrates example apparatus for processing an inference according to the present invention;

[0023] Figure 2 illustrates processor utilisation for a pair of conventional DPUs against a pair of DPUs operating in accordance with the present invention; and

[0024] Figures 3A and 3B illustrates cache management for DPUs 3 and 4 of fig. 2.

[0025] DETAILED DESCRIPTION OF DRAWINGS

[0026] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0027] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0028] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0030] Referring to figure 1 , example apparatus is illustrated including a server 10 comprising a central processing unit (CPUe) and an array of Data Processing Units (DPUs). Each DPU provides hardware acceleration for the processing of LLM inferences under the control of CPUe. The suffix ‘e’ in CPUeis provided to convey that CPUeis external to the DPUs and to distinguish CPUe from the internal CPUs of the DPUs. CPUeis connected to memory (DRAM) and, via a serial PCI Express / Compute Express Link (PCIe / CXL), to the the DPUs and a hard drive (SSD). CXL is an open standard for high-speed, high capacity CPU-to-device and CPU-to- memory connections, and is designed for high performance data center computers. Although a PCIe / CXL interconnect is show, others high-speed interconnects may be used.

[0031] Fig. 1 also illustrates a DPU in greater detail comprising a CPU (CPU; with the suffix T denoting that the CPU is internal to the DPU), memory (DRAM) and a PCIe / CXL interface. DRAM. The DPU is further provided with a plurality of network processing units (NPUs) configured to perform massively parallel processing of inferences, under the control of CPU; of the DPU. I.e. hardware acceleration. DPU DRAM provides each DPU with a large memory including for caching KV data.

[0032] On a high level, cooperating pairs of DPU function to provide CPUeof the compute server with hardware acceleration for processing of inferences in accordance with the present invention including by prefilling to generate KV data; caching multiple copies of the KV data, including at least one copy at one DPU and at least a further copy at a second cooperating DPU. Depending on the workload and / or memory utilisation of the first and second processing nodes, the cached KV data can be decoded at either or both of the cooperating pairs of DPUs.

[0033] This can be illustrated by contrasting DPU utilisation of conventional DPUs for processing an inference against the same for a co-operating pair of DPUs operating in accordance with the present invention. More specifically, figure 2 illustrates processor utilisation U(#) against time T for a first pair P1 of conventional DPUs 1 and 2 against a second pair P2 of DPUs 3 and 4 operating in accordance with the present invention. The conventional pair of DPUs 1 and 2 is operating whereby DPU 1 exclusively performs the function of decoding of requests, and whereby DPU 2 exclusively performs the function of prefilling of those requests. In particular, utilisation U(1) of DPU 1 shows a serious of decoding periods D11 , D12, D21 , D22, D3 and D4 corresponding to active periods A1 and A2. Overall, DPU 1 is mostly active. Utilisation U(2) of DPU 2 shows a serious of prefilling periods for requests R1 to R4 corresponding to collective active periods A3 to A6. Overall, DPU 2 is mostly inactive.

[0034] Certain observations can be made.

[0035] Prefilling by DPU 2 of first request R1 must be completed before decoding of that request by DPU 1 can be begin (at the end of time period A3). Decoding of the first request is done in periods D11 and D12, i.e. at different rates of utilisation, resulting in the completion of the decoding of the first request R1 at time E1 . During decoding of the first request R1 by DPU 1 , DPU 2 prefills a second request at R2. Decoding of the second request R2 is done in periods D21 and D22, again at different rates of utilisation, resulting in the completion of the decoding of the second request R2 at time E2. During DPU 1’s decoding of the second request R2, DPU 2 prefills a third request at R3. Decoding of the third request R3 is done in period D3. As illustrated, DPU 1 is briefly inactive having decoded the first R1 , second R2 and third R3 requests before the completion of prefilling by DPU 2 of a fourth request R4.

[0036] Contrast this with the processor utilisation U(3) and U(4) for a pair of DPUs 3 and 4 operating in accordance with the present invention. As with DPU 2, DPU 4 is wholly responsible for prefilling of requests R1 to R4 which as illustrated are presented to the DPUs at the same time. Prefilling by DPU 4 of first request R1 must be completed before decoding of that request by DPU 3 and 4 can be begin. After prefilling of request R1 , both DPUs 3 and 4 decode request R1 in periods D11 to D14, sharing the workload and resulting in the completion of the decoding of the first request R1 at time E1 , AT1 faster than with DPUs 1 and 2. Similarly, when processing second request R2, DPUs 3 and 4 decode in periods D21 to D24. resulting in the completion of the decoding of the second request R2 at time E2, AT2 faster than with DPUs 1 and 2. This is because of utilisation of spare decoding capacity of DPU 4 when it is not prefilling, increasing the maximum decoding capacity at any given time.

[0037] DPUs 3 and 4 do not have to share the decoding of each request. For example, as with DPU 1 , third request R3 is processed only by DPU 3 since only a modest amount of processing is required which may not be sufficient to justify the overhead of sharing. As illustrate, DPU 3 is inactive at time E3 after processing request R3 until the prefilling of request R4.

[0038] To facilitate the sharing of decoding, copies of the KV data must be cached, including at least one copy for in a cache of each of DPU 3 and DPU 4. Cache management and cache synchronization of DPUs 3 and 4 is illustrated in figures 3A and 3B.

[0039] In figure 3A, DPU 3 cache 30’ and DPU 4 cache 30 are shown at a time, with reference to figure 2, where DPU 4 is prefilling KV data from request R2 - indicated by ‘PF KV (R2)’ - and DPU 3 is decoding KV data from request R1 during period D11 - indicated by ‘D1 T. At the same time, a copy of the KV data from the cache of DPU 4 is transmitted to the cache of DPU 3 30’. Also, a copy of the KV data decoded by DPU 3 from request R1 during period D11 is transferred to DPU 4 cache 30. More specifically, newly computed KV cache lines are transferred to ensure synchronization.

[0040] In figure 3B, DPU 3 cache 30’ and DPU 4 cache 30 are shown at a time, with reference to figure 2, where both DPU 3 and 4 are decoding KV data from request R2 during respective D21 and D22 (immediately after time E1 shown in fig. 2). At the same time, a copy of the KV data decoded by either DPU is transmitted to the other. Again, more specifically, newly computed KV cache lines are transferred to ensure synchronization.

[0041] As well as the potential for faster decoding (AT1 and AT2 in figure 2), sharing of decoding capacity can also enable load balancing so that one DPU is not overloaded compared to another. For example, as illustrated in fig. 2, the utilisation of DPU 1 for decoding request R1 in periods D11 and D12 is approximately twice that of the decoding request R1 in periods D11 to D14 by DPUs 3 and 4 together. Lowering the maximum utilisation of a DPU when performing decoding may reduce power consumption and prolong the life of such a DPU.

[0042] In the above example, a pair of DPUs is utilised, though of course a large cooperating group of DPUs and / or other processing nodes could be utilised. Also, although the invention is described with DPUs, other types of processing nodes may also be used.

[0043] The present invention leverages redundant KV cache data to split prefill and decoding requests. As evidenced by performance modelling, this approach can reduce latency and increase throughput through load balancing and managing workload. This can be perceived by the user in terms of time to first token (TTFT), i.e. the latency of prefilling. Also, this can also be perceived by the user in terms of time between time between tokens (TBT), i.e. the latency of decoding. With multiple decoding DPUs, a lower time between tokens can be obtained. For example, next tokens may identify the most likely word or phrase to follow a natural language input, and thus provide rapid text auto-completion, speech recognition, machine translation and the like, providing a smooth and predictable rate of output for easy reading.

[0044] Also, whilst the LLM illustrated above is used with a text input, the invention may also apply to a video and / or audio input to an LLM for a text output.

[0045] The present invention would typically be used in Al server environments where efficient and memory and resource management are critical, and DPUs providing acceleration.

[0046] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive.

Claims

CLAIMS1. A method of processing an inference of a large language model, LLM, comprising: prefilling to generate key value, KV, data; caching multiple copies of the KV data, including at least one copy at a first processing node and at least a further copy at a second processing node; and depending on the workload and / or memory utilisation of the first and second processing nodes, decoding the cached KV data at either the first processing node or the second processing node.

2. A method according to claim 1 , wherein copies of the cached KV data are synchronised.

3. A method according to claim 1 or claim 2, wherein one processing node is solely responsible for prefilling.

4. A method according to claim 3, wherein the one processing node solely responsible for prefilling also shares decoding with the other processing node(s) to reduce idling when not prefilling.

5. A method according to any of the preceding claims, wherein: a plurality of pairs of processing nodes are provided, wherein, each pair of processing nodes processes an inference whereby copies of cached KV data are synchronised between processing nodes of each pair of processing nodes.

6. A method according to claim 5, further comprising: upon completion of the processing of an inference by one pair of processing nodes, transferring copies of cached KV data to another pair of processing nodes for processing a further inference using the transferred cached KV data.

7. Apparatus comprising at least first and second processing nodes, the apparatus being configured to perform a method as claimed in any of the proceeding claims.

8. Apparatus according to claim 7, wherein the at least first and second processing nodes each comprise: a data processing unit, DPU, configured to provide hardware acceleration for decoding in the processing of an inference of a LLM,8wherein each DPU comprises a respective cache.

9. Apparatus according to claim 8 further comprising: a host processor configured to control the DPUs of the processing nodes and determine, depending on the workload and / or memory utilisation of the first and second processing nodes, which processing node is to process KV data from its respective cache.

10. Apparatus according to claim 8 or claim 9 wherein copies of the cached KV data in the respective caches of the processing nodes are synchronised.

11. Apparatus according to any of claims 7 to 10 wherein one processing node is solely responsible for prefilling.

12. Apparatus according to claim 11, wherein the one processing node solely responsible for prefilling also shares decoding with the other processing nodes to reduce idling when not prefilling.

13. Apparatus according to any of claims 7 to 12, wherein a plurality of pairs of processing nodes are provided, wherein each pair of processing nodes is configured to process an inference whereby copies of cached KV data are synchronised between processing nodes of each pair of processing nodes.

14. Apparatus according to any of claim 13, further configured to, upon completion of the processing of an inference by one pair of processing nodes, transfer copies of cached KV data to another pair of processing nodes for processing a further inference using the transferred KV data.

15. A DPU of apparatus according to any of claims 8 to 14.9

Images