LLM inference method and heterogeneous system based on GPU and LPU

Abstract

The application discloses a GPU and LPU-based LLM inference method and a heterogeneous system, the method is applied to a host server provided with a GPU computing unit, an LPU computing unit composed of multiple LPU cards and an RDMA communication network, and comprises the following steps: a large language model is split and deployed according to operator types, attention-related operators are deployed on the GPU computing unit, and feedforward network layer weights are preloaded to on-board memories of the LPU cards according to a splitting strategy; the GPU computing unit performs attention calculation to generate attention intermediate results, and the attention intermediate results are directly written into on-board memories connected with LPU side network cards through optical fibers in a remote direct memory access mode without passing through a central processor and a system memory of the host; each LPU card performs feedforward network calculation in parallel based on matched input data fragments and weight fragments, and returns output fragments, and the inference results are output layer by layer in a circulation mode after being gathered by the GPU computing unit. The application reduces single GPU memory requirements and hardware costs, improves decoding throughput, and avoids communication overhead of the host.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing and artificial intelligence inference acceleration technology, specifically to heterogeneous computing and distributed inference, and particularly to an LLM inference method and heterogeneous system based on GPU and LPU. Background Technology

[0002] As the parameter scale of large language models expands from billions to hundreds of billions or even trillions, their inference deployment faces severe challenges in terms of computation, storage, and cost. The inference process of large language models typically includes two stages: pre-filling and decoding. These two stages have significantly different computational characteristics: the pre-filling stage encodes the input word sequence in parallel, a computationally intensive operation that fully utilizes the computing power of the massively parallel computing units in the graphics processing unit (GPU); the decoding stage generates output word by word in an autoregressive manner, involving only a small number of matrix operations each time, making it a memory-intensive operation. During this stage, the GPU's parallel computing units are mostly idle, resulting in low actual computational utilization, and system throughput is mainly limited by GPU memory bandwidth. In related technologies, pre-filling and decoding are often deployed on the same GPU cluster, leading to a significant amount of GPU computing power being underutilized during the decoding stage.

[0003] On the other hand, the memory capacity of a single graphics processing unit (GPU) is limited, making it difficult to accommodate all the parameters of a large language model, among which the weights of the feedforward network layer account for the largest proportion of the model parameters. To accommodate the model parameters, related technologies often use tensor parallelism or pipelined parallelism to divide the model and deploy it on multiple high-capacity GPUs, which leads to high hardware costs. In scenarios where only the decoding throughput needs to be improved, the cost-effectiveness of configuring additional high-capacity GPUs is low.

[0004] To alleviate these issues, related technologies have attempted to offload some computation to CPU memory or solid-state drives (SSDs). However, the former is limited by computing power and memory bandwidth, while the latter is limited by storage access latency, both of which struggle to meet the real-time requirements of inference. Some solutions attempt to accelerate inference using dedicated processors, but these mostly focus on operator acceleration on a single board, lacking a complete solution for efficient collaboration between graphics processing units (GPUs) and multiple dedicated processing cards. Furthermore, if data transfer between heterogeneous devices is relayed through host memory, it introduces significant communication latency and CPU overhead, potentially offsetting the benefits of computation offloading. In summary, related technologies in large language model inference suffer from technical problems such as limited single-GPU memory capacity, low computational utilization during the decoding stage, high system hardware costs, and insufficient communication efficiency between heterogeneous devices, necessitating the development of new solutions. Summary of the Invention

[0005] To address the shortcomings of the existing technology, this application provides an LLM inference method and heterogeneous system based on GPU and LPU. The technical effects to be achieved by this application are realized through the following scheme: The first aspect of this application: A GPU- and LPU-based LLM inference method is provided, applied to a host server deployed with GPU computing units, LPU computing units composed of multiple LPU cards, and an RDMA communication network, including the following steps: The large language model is split and deployed according to operator type. Attention-related operators are deployed on the GPU computing unit. The weights of the feedforward network layer are preloaded to the onboard memory of each LPU card according to the splitting strategy to obtain the weight slices of each LPU card. The GPU computing unit performs attention calculations on the input data of the current network layer to generate intermediate attention results. The attention intermediate results are written directly to the onboard memory connected to the LPU-side network card via a fiber optic link using remote direct memory access via the GPU-side network card. The LPU-side network interface card distributes the attention intermediate results into input data fragments that match each of the weight fragments according to the segmentation strategy; Each LPU card performs feedforward network computation in parallel based on its corresponding input data slice and weight slice to obtain the feedforward network output slice; Each LPU card transmits the feedforward network output fragments back to the GPU computing unit via the LPU-side network card using remote direct memory access. The GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, completes the calculation of the current network layer, uses the feedforward network output of the current network layer as the input of the next network layer, and cyclically executes attention calculation and subsequent steps until all network layers are completed and the inference result is output.

[0006] Preferably, the weights of the feedforward network layer are preloaded into the onboard memory of each LPU card according to a partitioning strategy, including: Based on the characteristic that the weights of the feedforward network layer account for the largest proportion, the weights of the feedforward network layer are divided into multiple weight slices according to the tensor parallel partitioning method or the pipeline parallel partitioning method, and are preloaded and resided in the onboard memory of each LPU card. The GPU computing unit stores only the parameters of the remaining layers in the large language model, excluding the feedforward network layer.

[0007] Preferably, performing attention calculation on the input data of the current network layer includes: The attention-related operators include embedding operators, attention operators, and normalization operators; During the pre-filling phase, the GPU computing unit performs the attention calculation on the input word sequence in parallel; During the decoding phase, the GPU computing unit performs the attention calculation word by word in an autoregressive manner, and the resulting intermediate attention result is a hidden state tensor.

[0008] Preferably, the GPU-side network interface card (NIC) uses remote direct memory access (RDBMI) to directly write the attention intermediate results into the onboard memory connected to the LPU-side NIC via a fiber optic link, including: The GPU-side network card and the LPU-side network card are physically connected via optical modules and optical fibers to form a direct data link between the domain where the GPU computing unit is located and the domain where the LPU computing unit is located. Through the direct data link, the GPU-side network card directly reads and writes data in the onboard memory of the LPU card, and the LPU-side network card directly reads and writes data in the video memory of the GPU computing unit, and the writing is a zero-copy write without data relay copying.

[0009] Preferably, the LPU-side network interface card distributes the intermediate attention results into input data fragments that match each of the weighted fragments according to the segmentation strategy, including: The LPU-side network interface card distributes the attention intermediate results to each LPU card in broadcast mode or point-to-point mode, so that the input data fragments obtained by each LPU card match the weight fragments residing in that LPU card. The LPU cards exchange intermediate fragment data required for parallel execution of feedforward network calculations via direct optical connection channels.

[0010] Preferably, each LPU card performs feedforward network computation in parallel based on its corresponding input data slices and weight slices, including: Each LPU card performs the feedforward network computation through a fixed computational structure consisting of vector and matrix multiplication units, activation operator units, and residual addition units; The feedforward network computation is a gated computation, comprising: dividing the input data into pieces and subjecting them to a first linear transformation and a second linear transformation respectively; processing the result of the first linear transformation with an activation operator and multiplying it element-wise with the result of the second linear transformation to obtain an intermediate tensor; subjecting the intermediate tensor to a third linear transformation and superimposing the residuals to obtain the output pieces of the feedforward network.

[0011] Preferably, each of the LPU cards transmits the feedforward network output fragments back to the GPU computing unit via the LPU-side network card using remote direct memory access, including: Each LPU card first aggregates its feedforward network output fragments into data to be transmitted back within the LPU computing unit via the optical port direct connection channel; The data to be transmitted is then written directly into the GPU computing unit's memory via the LPU-side network card using remote direct memory access.

[0012] Preferably, the GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, thereby completing the calculation of the current network layer and using it as the input of the next network layer. This process is repeated until all network layers are completed and the inference result is output, including: After the GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, it performs a normalization operation on the feedforward network output to complete the calculation of the current network layer. The large language model is a multi-layered stacked Transformer structure. The calculation result of the current network layer is used as the input of the next network layer. The attention calculation, feedforward network calculation and normalization operation are performed layer by layer in a loop until the calculation of all network layers is completed and the inference result is output.

[0013] Preferably, the method further includes heterogeneously splitting the computational tasks of the decoding stage: During the pre-filling stage of the large language model inference, the GPU computing unit performs parallel encoding computation on the input word sequence; During the decoding stage of the large language model inference, the decoding computation task of word-by-word generation is proportionally split into the GPU computing unit and the LPU computing unit for parallel execution, so that the GPU computing unit and the LPU computing unit can concurrently process multiple decoding requests, and the LPU computing unit supplements the storage capacity and memory access bandwidth required for the decoding stage.

[0014] The second aspect of this application: A heterogeneous LLM inference system based on GPU and LPU is provided, deployed on a host server, including: The GPU computing unit is configured to perform attention calculations on the input data of the current network layer and generate intermediate attention results, as well as to aggregate the feedforward network output slices to obtain the feedforward network output of the current network layer and loop to the next network layer. An LPU computing unit composed of multiple LPU cards has onboard memory of each LPU card pre-reserving weight slices of the feedforward network layer, which are configured to perform feedforward network computation in parallel based on matching input data slices and the weight slices to obtain feedforward network output slices. The RDMA communication network includes a GPU-side network card connected to the GPU computing unit and an LPU-side network card connected to the LPU computing unit. The two are connected via a fiber optic link and are configured to directly transmit the attention intermediate results and the feedforward network output fragments between the GPU computing unit and the LPU computing unit via remote direct memory access, and the transmission does not pass through the central processing unit and system memory of the host server.

[0015] Compared with related technologies, the present invention has the following beneficial effects: Firstly, it reduces system hardware costs. This invention completely offloads the feedforward network layer weights, which account for the largest proportion of parameters, to the onboard memory of multiple LPU cards for storage and computation. This allows a single graphics processor to store only the attention-related operators and other layer parameters, thereby significantly reducing the video memory requirements of a single graphics processor, thus reducing the number of high-capacity graphics processors needed and lowering the system hardware procurement costs.

[0016] Secondly, it improves inference throughput during the decoding stage. This invention splits the computation of the decoding stage onto the graphics processor and multiple LPU cards for parallel execution. The LPU cards supplement the additional storage capacity and memory access bandwidth, so that the decoding process is no longer completely limited by the graphics processor's video memory bandwidth. The system can process multiple decoding requests concurrently, thereby improving the overall inference throughput.

[0017] Third, it reduces the generation cost of individual words. This invention enables computationally intensive tasks to be handled by graphics processors and memory-intensive tasks by more energy-efficient LPU cards, achieving a reasonable match of heterogeneous computing resources. This results in higher throughput with the same hardware investment, or lower hardware investment with the same throughput requirements, thereby reducing the average generation cost of a single word.

[0018] Fourth, it avoids host relay latency and achieves efficient heterogeneous collaborative communication. This invention transmits data directly between the graphics processor and the LPU card through remote direct memory access technology. Data does not need to be relayed through the central processing unit and system memory of the host server, avoiding the data copy latency and central processing unit overhead introduced by related communication methods. This ensures the real-time performance of frequent data interactions across devices under the attention and feedforward network separation computing architecture, so that the benefits of computing offloading are not offset by communication overhead. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the LLM inference method based on GPU and LPU according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the configuration scheme of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the configuration scheme of Embodiment 2 of the present invention; Figure 4 This is a structural block diagram of a heterogeneous system according to an embodiment of the present invention. Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This embodiment provides an LLM inference method and heterogeneous system based on GPU and LPU.

[0022] The GPU computing unit consists of one or more graphics processing unit accelerator cards, suitable for computationally intensive parallel computing; the LPU computing unit consists of multiple dedicated processing cards, each LPU card being a dedicated computing chip for feedforward network operators with lower computing power and higher memory access bandwidth compared to the graphics processor, suitable for memory-intensive feedforward network computing; the RDMA communication network is used to establish a low-latency, high-bandwidth direct data channel between the above two types of computing units. The hardware architecture of the system is described first, followed by four embodiments.

[0023] System hardware architecture: The GPU and LPU multi-card heterogeneous inference system of this invention is deployed on a multi-slot host server. The host server is connected to three hardware components—GPU computing unit, LPU computing unit, and RDMA communication network—through a high-speed peripheral interconnect interface.

[0024] A GPU computing unit consists of one or more graphics processing unit accelerator cards, each equipped with high-bandwidth memory as video memory.

[0025] In one example, the high-bandwidth video memory capacity of the graphics processing unit accelerator card used is approximately 80GB, and the video memory bandwidth is approximately 3.4TB / s (the above values ​​are example values ​​and do not constitute a limitation of the present invention).

[0026] The GPU computing unit is primarily responsible for calculating attention-related operators, and in some configurations, it also handles some of the computations during the decoding stage. Each graphics processing unit accelerator card is connected to the host server motherboard via a peripheral interconnect interface and directly connected to the side network card via an onboard high-speed interconnect interface.

[0027] The LPU computing unit consists of multiple LPU accelerator cards, each equipped with onboard high-bandwidth memory.

[0028] In one example, the LPU accelerator card used has an onboard memory capacity of approximately 40GB and a memory access bandwidth of approximately 1TB / s. Its single-card cost is approximately one-tenth of the aforementioned graphics processor accelerator card (the above values ​​are example values ​​and do not constitute a limitation of the present invention).

[0029] The weight parameters of the feedforward network layer are pre-loaded and continuously reside in the onboard high-bandwidth memory of the LPU accelerator card, thus avoiding repeated loading of weights from external storage during runtime. Each LPU accelerator card is connected to the host server motherboard via a peripheral interconnect interface and directly connected to the LPU-side network card via an onboard high-speed interconnect interface.

[0030] The RDMA communication network is the core data path connecting the GPU computing unit and the LPU computing unit. It includes a multi-port GPU-side network card that supports the Remote Direct Memory Access protocol and a multi-port LPU-side network card that supports the Remote Direct Memory Access protocol.

[0031] In one example, the network card's single-port bandwidth is approximately 400Gbps, and each LPU accelerator card also provides a direct optical connection channel with a bandwidth of approximately 200Gbps (the above bandwidths are example values ​​and do not constitute a limitation of the present invention). The GPU-side network card and each graphics processor accelerator card, as well as the LPU-side network card and each LPU accelerator card, are internally interconnected via onboard high-speed buses. The GPU-side network card and the LPU-side network card are physically connected via optical modules and fiber optic links, forming a high-speed direct data link between the GPU computing unit domain and the LPU computing unit domain. This data link supports remote direct memory access operations; the GPU-side network card can directly read and write data in the LPU accelerator card's onboard memory, and the LPU-side network card can also directly read and write data in the graphics processor accelerator card's video memory. The entire data transmission process does not require intervention from the host server's central processing unit, nor does it require relaying through system memory, thereby achieving zero-copy data transmission between the graphics processor's video memory and the LPU's onboard memory, reducing communication latency and system overhead to a low level.

[0032] Example 1, First configuration scheme: This embodiment is suitable for application scenarios that are sensitive to system hardware costs and have moderate inference throughput requirements. Its hardware configuration consists of one graphics processing unit (GPU) accelerator card and four LPU accelerator cards. Figure 2 As shown.

[0033] The reasoning method in this embodiment includes steps S1 to S7, see [link / reference] Figure 1 .

[0034] Step S1: Split and deploy the large language model according to operator type.

[0035] Specifically, attention-related operators are deployed on GPU computing units, including embedding operators, attention operators, and normalization operators; the weights of the feedforward network layer are preloaded into the onboard memory of the four LPU accelerator cards according to a splitting strategy to obtain weight slices for each LPU card.

[0036] In this embodiment, based on the characteristic that the feedforward network layer has the largest weight ratio, it is completely offloaded to the LPU computing unit, while the GPU computing unit only needs to store the parameters of the other layers besides the feedforward network layer.

[0037] The segmentation strategy can adopt a tensor parallel segmentation method, which divides the same weight matrix of the feedforward network layer into several pieces along a specific dimension and stores them in different LPU cards; or it can adopt a pipelined parallel segmentation method, which distributes the feedforward network weights of different network layers to different LPU cards.

[0038] The selection of the partitioning strategy is based on the model structure and inference latency requirements. When the latency of a single layer is more sensitive, the tensor parallel partitioning method is preferred, while the pipeline parallel partitioning method is preferred when the number of network layers is large and throughput is pursued. The specific number of partitions and the dimension of each partition are adjusted according to the number of LPU cards and their onboard memory capacity to make the storage usage of each LPU card relatively balanced.

[0039] In step S2, the GPU computing unit performs attention calculations on the input data of the current network layer, generating intermediate attention results. During the pre-filling stage, the GPU computing unit performs embedding, attention, and normalization calculations in parallel on the input word sequence; during the decoding stage, the GPU computing unit performs these calculations word-by-word in an autoregressive manner. The generated intermediate attention results are hidden state tensors, which are passed to step S3 as input data for subsequent feedforward network calculations.

[0040] In step S3, the attention intermediate results generated in step S2 are directly written to the onboard memory connected to the LPU side network card via a fiber optic link using remote direct memory access through the GPU-side network card, without passing through the host server's central processing unit and system memory. Since the GPU-side network card and the LPU-side network card are physically connected via optical modules and fiber optics to form a direct data link, the attention intermediate results are written directly from the graphics processor's memory to the LPU accelerator card's onboard memory in a zero-copy manner, without first copying to system memory and then forwarding. This significantly reduces cross-domain transmission latency and central processing unit overhead.

[0041] In step S4, the LPU-side network interface card (NIC) distributes the intermediate attention results written in step S3 into input data fragments that match the weight fragments, according to the partitioning strategy determined in step S1. Specifically, the LPU-side NIC distributes the intermediate attention results to each LPU accelerator card in either broadcast or point-to-point mode, ensuring that the input data fragments obtained by each LPU card match the weight fragments residing on that card in terms of dimension. When using tensor parallel partitioning, broadcast mode is preferred because each fragment computation needs to share the same input; when using pipelined parallel partitioning, point-to-point mode is preferred because the data of each network layer has sequential dependencies. Furthermore, the LPU accelerator cards can exchange intermediate fragment data required for parallel execution of feedforward network computation via direct optical connection channels to support cross-card tensor parallel aggregation.

[0042] In step S5, each LPU accelerator card performs feedforward network computation in parallel based on the input data slices corresponding to its own card obtained in step S4 and the weight slices residing in step S1, to obtain the feedforward network output slices. Each LPU accelerator card performs feedforward network computation through a fixed computational structure consisting of vector and matrix multiplication units, activation operator units, and residual addition units.

[0043] In this embodiment, the feedforward network employs a gated computation structure, which includes a first linear transformation unit, an activation operator unit, a second linear transformation unit, an element-wise multiplication unit, and an output linear transformation unit. The computation process is as follows: the input data is segmented and subjected to the first and second linear transformations respectively. The result of the first linear transformation is processed by the activation operator and then multiplied element-wise with the result of the second linear transformation to obtain an intermediate tensor. This intermediate tensor is then subjected to a third linear transformation and the residuals are superimposed to obtain the feedforward network output segments.

[0044] In step S6, each LPU accelerator card transmits the feedforward network output fragments obtained in step S5 back to the GPU computing unit via the LPU-side network card using remote direct memory access.

[0045] In one implementation, each LPU accelerator card first aggregates its feedforward network output fragments into data to be transmitted back within the LPU computing unit via a direct optical port connection channel, and then writes the data to be transmitted back directly into the graphics processor accelerator card's memory in a zero-copy manner via the LPU-side network card. In another implementation, each LPU accelerator card can also send its own output fragments back directly without aggregation, and the GPU computing unit will aggregate them in step S7.

[0046] The former implementation can reduce the amount of data transmitted back and the number of network card transmissions, while the latter implementation can reduce the synchronization overhead within the LPU domain. The choice can be made based on the network layer dimension and the inter-card bandwidth.

[0047] In step S7, the GPU computing unit aggregates the feedforward network output fragments returned in step S6 to obtain the feedforward network output of the current network layer, and completes the calculation of the current network layer accordingly.

[0048] Specifically, after the GPU computing unit gathers the output of the feedforward network, it performs a normalization operation on it to complete the calculation of the current network layer. Since the large language model is a multi-layer stacked Transformer structure, the GPU computing unit uses the calculation result of the current network layer as the input of the next network layer, and returns to step S2 to perform attention calculation, feedforward network calculation and normalization operation layer by layer until the calculation of all network layers is completed and the inference result is output.

[0049] Through the above steps S1 to S7, this embodiment completely offloads the weights of the feedforward network layer with the largest parameter proportion to the four LPU accelerator cards, so that a single graphics processor only needs to store the other layer parameters such as attention-related operators, which greatly reduces the memory requirements of a single graphics processor.

[0050] Large model parameters that would normally require two or more graphics processors can be deployed in this embodiment with just one graphics processor and four LPU cards, thus effectively reducing system hardware procurement costs.

[0051] Taking a large language model with approximately 230 billion parameters as an example, under a quantized data format with four-bit weights and eight-bit activation, when the context length is 64K, the pure graphics processor solution using related technologies requires at least two of the aforementioned graphics processor accelerator cards. However, the solution using this embodiment only requires one graphics processor accelerator card and four LPU accelerator cards to complete the deployment. Actual test data shows that the system hardware cost is reduced by approximately 40% (this value is an exemplary test result and is only used for illustration, and does not constitute a limitation of the present invention).

[0052] Example 2, Second Configuration Scheme: This embodiment is suitable for large-scale production deployment scenarios with high requirements for inference throughput and latency. Its hardware configuration consists of two graphics processing unit (GPU) accelerator cards and eight LPU accelerator cards, such as... Figure 3 As shown.

[0053] The difference from Embodiment 1 is that this embodiment performs heterogeneous splitting and parallelization of the computational tasks in the decoding stage.

[0054] In this embodiment, the weights of the feedforward network layer are distributed and stored in the onboard memory of eight LPU accelerator cards using a tensor parallel partitioning method. During the pre-filling stage, due to its computationally intensive nature, the parallel encoding calculation of the input word sequence is still mainly completed collaboratively by two graphics processing unit accelerator cards. During the decoding phase, the decoding computation task of word-by-word generation is proportionally split into GPU computing units and LPU computing units for parallel execution. That is, two graphics processing unit accelerator cards undertake part of the decoding request computation, and eight LPU accelerator cards undertake the other part of the decoding request computation, and the two are carried out concurrently.

[0055] The split ratio is set based on the ratio of the effective memory access bandwidth of the graphics processor and the LPU card, so that the decoding load allocated to the two types of computing units matches their memory access capabilities, thereby avoiding one type of computing unit from becoming a bottleneck.

[0056] During decoding, the GPU computing unit and the LPU computing unit synchronize data and exchange intermediate results via a high-speed interconnect channel established by the two network cards. Because the LPU computing unit provides additional storage capacity and memory access bandwidth, the computation during the decoding stage is no longer entirely limited by the GPU's memory bandwidth. The total effective memory access bandwidth provided by the GPU and LPU cards in collaboration is higher than that of a pure GPU solution, allowing the system to handle more decoding requests concurrently and improving overall inference throughput.

[0057] Taking the aforementioned large language model with approximately 230 billion parameters as an example, under the conditions of four-bit weights, eight-bit activation quantized data format, and a context length of 64K, according to the actual test data provided by the inventors, the decoding throughput of the pure graphics processor solution using two graphics processor accelerator cards is approximately 800 words per second, while the decoding throughput of the solution using two graphics processor accelerator cards in combination with eight LPU accelerator cards in this embodiment is approximately 1200 words per second, with a throughput increase of approximately 50% (the above values ​​are exemplary test results and are only used for illustration, and do not constitute a limitation of the present invention).

[0058] Example 3: Flexible expansion of configuration schemes.

[0059] The hardware configurations of the two embodiments described above are merely illustrative and are not intended to limit the scope of the invention.

[0060] The configuration ratio of GPU computing units to LPU computing units can be flexibly adjusted according to the differences in the number of parameters of the model to be deployed, the inference throughput requirements, and the system cost budget.

[0061] For example, in scenarios with larger parameter counts or lower cost budgets, the system can be expanded to a configuration of one GPU accelerator card and eight LPU accelerator cards to further reduce the memory pressure on a single GPU. For scenarios with higher throughput requirements, the system can be expanded to a configuration of four GPU accelerator cards and eight LPU accelerator cards, or eight GPU accelerator cards and sixteen LPU accelerator cards, increasing the number of these two types of computing cards to expand system throughput. The distribution of feedforward network layer weights across multiple LPU cards can be achieved using either tensor-parallel partitioning or pipeline-parallel partitioning.

[0062] Furthermore, the deployment method of the model and the allocation strategy of computing tasks can be dynamically adjusted through configuration files and runtime management programs. System administrators can dynamically adjust the task allocation ratio between the two types of computing units based on the actual load without modifying the hardware topology or restarting the system, so as to achieve the rational utilization of system resources.

[0063] Example 4: A GPU and LPU LLM inference heterogeneous system corresponding to the above method.

[0064] The system is deployed on a host server and includes a GPU computing unit, an LPU computing unit, and an RDMA communication network.

[0065] The GPU computing unit is configured to perform attention calculation on the input data of the current network layer and generate intermediate attention results, and to aggregate the feedforward network output slices to obtain the feedforward network output of the current network layer and loop to the next network layer, that is, to perform the above steps S2 and S7. The onboard memory of each LPU accelerator card of the LPU computing unit pre-resides the weight slices of the feedforward network layer, and is configured to perform feedforward network calculations in parallel based on the matching input data slices and weight slices to obtain the feedforward network output slices, which corresponds to the execution of the above step S5. The RDMA communication network includes a GPU-side network card and an LPU-side network card, which are connected via a fiber optic link. It is configured to directly transmit attention intermediate results and feedforward network output fragments between the GPU computing unit and the LPU computing unit via remote direct memory access, and the transmission does not pass through the central processing unit and system memory of the host server, which corresponds to the execution of the above steps S3, S4 and S6.

[0066] The specific working process of each of the above units can be found in the corresponding steps in the aforementioned method embodiments, and will not be repeated here.

[0067] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0068] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0069] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0070] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0071] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0072] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A GPU and LPU-based LLM inference method applied to a host server deployed with a GPU computing unit, an LPU computing unit composed of multiple LPU cards, and an RDMA communication network, characterized in that, Includes the following steps: The large language model is split and deployed according to operator type. Attention-related operators are deployed on the GPU computing unit. The weights of the feedforward network layer are preloaded to the onboard memory of each LPU card according to the splitting strategy to obtain the weight slices of each LPU card. The GPU computing unit performs attention calculations on the input data of the current network layer to generate intermediate attention results. The attention intermediate results are written directly to the onboard memory connected to the LPU-side network card via a fiber optic link using remote direct memory access via the GPU-side network card. The LPU-side network interface card distributes the attention intermediate results into input data fragments that match each of the weight fragments according to the segmentation strategy; Each LPU card performs feedforward network computation in parallel based on its corresponding input data slice and weight slice to obtain the feedforward network output slice; Each LPU card transmits the feedforward network output fragments back to the GPU computing unit via the LPU-side network card using remote direct memory access. The GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, completes the calculation of the current network layer, uses the feedforward network output of the current network layer as the input of the next network layer, and cyclically executes attention calculation and subsequent steps until all network layers are completed and the inference result is output.

2. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The weights of the feedforward network layer are preloaded into the onboard memory of each LPU card according to a partitioning strategy, including: Based on the characteristic that the weights of the feedforward network layer account for the largest proportion, the weights of the feedforward network layer are divided into multiple weight slices according to the tensor parallel partitioning method or the pipeline parallel partitioning method, and are preloaded and resided in the onboard memory of each LPU card. The GPU computing unit stores only the parameters of the remaining layers in the large language model, excluding the feedforward network layer.

3. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The process of performing attention calculations on the input data of the current network layer includes: The attention-related operators include embedding operators, attention operators, and normalization operators; During the pre-filling phase, the GPU computing unit performs the attention calculation on the input word sequence in parallel; During the decoding phase, the GPU computing unit performs the attention calculation word by word in an autoregressive manner, and the resulting intermediate attention result is a hidden state tensor.

4. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The GPU-side network interface card (NIC) uses remote direct memory access (RDBMI) to directly write the attention intermediate results to the onboard memory connected to the LPU-side NIC via a fiber optic link, including: The GPU-side network card and the LPU-side network card are physically connected via optical modules and optical fibers to form a direct data link between the domain where the GPU computing unit is located and the domain where the LPU computing unit is located. Through the direct data link, the GPU-side network card directly reads and writes data in the onboard memory of the LPU card, and the LPU-side network card directly reads and writes data in the video memory of the GPU computing unit, and the writing is a zero-copy write without data relay copying.

5. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The LPU-side network interface card distributes the intermediate attention results into input data fragments that match the weight fragments according to the segmentation strategy, including: The LPU-side network interface card distributes the attention intermediate results to each LPU card in broadcast mode or point-to-point mode, so that the input data fragments obtained by each LPU card match the weight fragments residing in that LPU card. The LPU cards exchange intermediate fragment data required for parallel feedforward network computation via direct optical connection channels.

6. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, Each LPU card performs feedforward network computation in parallel based on its corresponding input data slices and weight slices, including: Each LPU card performs the feedforward network computation through a fixed computational structure consisting of vector and matrix multiplication units, activation operator units, and residual addition units; The feedforward network computation is a gated computation, comprising: dividing the input data into pieces and subjecting them to a first linear transformation and a second linear transformation respectively; processing the result of the first linear transformation with an activation operator and multiplying it element-wise with the result of the second linear transformation to obtain an intermediate tensor; subjecting the intermediate tensor to a third linear transformation and superimposing the residuals to obtain the output pieces of the feedforward network.

7. The LLM inference method based on GPU and LPU according to claim 5, characterized in that, Each of the LPU cards transmits the feedforward network output fragments back to the GPU computing unit via the LPU-side network card using remote direct memory access, including: Each LPU card first aggregates its feedforward network output fragments into data to be transmitted back within the LPU computing unit via the optical port direct connection channel; The data to be transmitted is then written directly into the GPU computing unit's memory via the LPU-side network card using remote direct memory access.

8. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, thereby completing the computation of the current network layer and using it as the input for the next network layer. This process is repeated until all network layers are completed and the inference result is output, including: After the GPU computing unit aggregates the output fragments of each feedforward network to obtain the feedforward network output of the current network layer, it performs a normalization operation on the feedforward network output to complete the calculation of the current network layer. The large language model is a multi-layered stacked Transformer structure. The calculation result of the current network layer is used as the input of the next network layer. The attention calculation, feedforward network calculation and normalization operation are performed layer by layer in a loop until the calculation of all network layers is completed and the inference result is output.

9. The LLM inference method based on GPU and LPU according to claim 1, characterized in that, The method also includes heterogeneously splitting the computational tasks in the decoding phase: During the pre-filling stage of the large language model inference, the GPU computing unit performs parallel encoding computation on the input word sequence. During the decoding stage of the large language model inference, the decoding computation task of word-by-word generation is proportionally split into the GPU computing unit and the LPU computing unit for parallel execution, so that the GPU computing unit and the LPU computing unit can concurrently process multiple decoding requests, and the LPU computing unit supplements the storage capacity and memory access bandwidth required for the decoding stage.

10. A heterogeneous LLM inference system based on GPU and LPU, deployed on a host server, characterized in that, include: The GPU computing unit is configured to perform attention calculations on the input data of the current network layer and generate intermediate attention results, as well as to aggregate the feedforward network output slices to obtain the feedforward network output of the current network layer and loop to the next network layer. An LPU computing unit composed of multiple LPU cards has onboard memory of each LPU card pre-reserving weight slices of the feedforward network layer, which are configured to perform feedforward network computation in parallel based on matching input data slices and the weight slices to obtain feedforward network output slices. The RDMA communication network includes a GPU-side network card connected to the GPU computing unit and an LPU-side network card connected to the LPU computing unit. The two are connected via a fiber optic link and are configured to directly transmit the attention intermediate results and the feedforward network output fragments between the GPU computing unit and the LPU computing unit via remote direct memory access, and the transmission does not pass through the central processing unit and system memory of the host server.

Images