Implementation method, device and equipment of large language model reasoning in heterogeneous system and medium

Abstract

The application discloses an implementation method and device of large language model reasoning in a heterogeneous system, equipment and a medium, wherein the heterogeneous system comprises a CPU, a GPU and an FPGA, the method comprises the following steps: performing attention calculation and key-value caching of the large language model through the GPU, performing routing calculation at an MoE layer, and performing shared expert multi-layer perception calculation to obtain a first execution result; constructing a routing expert execution request based on routing selection results and hidden states through the CPU, and submitting the routing expert execution request to the FPGA; performing multi-layer perception calculation of the activated routing experts according to the routing expert execution request through the FPGA to obtain a second execution result; and finally, fusing the second execution result with a shared expert output result through the GPU, and continuing to perform subsequent operations. The application can realize large language model reasoning through the cooperation of the CPU, the GPU and the FPGA, and effectively improves the resource utilization.

Description

Technical Field

[0001] This invention relates to the field of large language model reasoning technology, and in particular to a method, apparatus, device and medium for implementing large language model reasoning in heterogeneous systems. Background Technology

[0002] Mixture of Experts (MoE) is a commonly used deep learning framework in large language models. It can reduce the actual computational cost of reasoning in large language models by using multilayer perceptron (MLP) computation of sparse activation routing experts.

[0003] Currently, existing inference frameworks typically retain the MoE (Moment of Effect) within the graphics processing unit (GPU) software stack, with the GPU performing the entire process, including attention calculation, key-value caching, routing calculation, multilayer perceptron calculation for shared experts, and multilayer perceptron calculation for routing experts. However, calculating all routing experts within the GPU leads to significant GPU computational resource consumption.

[0004] Therefore, there is an urgent need for a large language model reasoning method that can improve resource utilization. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for implementing large language model reasoning in heterogeneous systems, which can improve resource utilization during large language model reasoning.

[0006] According to a first aspect of the present invention, a method for implementing large language model inference in a heterogeneous system is provided. The heterogeneous system includes a central processing unit (CPU), a GPU, and a field-programmable gate array (FPGA). A large language model is deployed on the CPU, and the large language model includes at least one hybrid expert MoE layer. The method includes:

[0007] The GPU performs attention calculation and key-value caching of the large language model, performs routing calculation at the MoE layer, and performs shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes information about the activated routing experts.

[0008] The CPU constructs a routing expert execution request based on the routing selection result and the hidden state, and submits the routing expert execution request to the FPGA;

[0009] The second execution result is obtained by performing multilayer perceptron calculations on the activated routing expert through the FPGA according to the routing expert execution request;

[0010] The second execution result is obtained through the GPU, and the second execution result is fused with the shared expert output result before continuing to execute subsequent operations.

[0011] According to a second aspect of the present invention, an implementation apparatus for large language model inference in a heterogeneous system is provided, the heterogeneous system including a CPU, a GPU, and an FPGA, wherein a large language model is deployed on the CPU, the large language model including at least one MoE layer, and the apparatus includes:

[0012] The first execution module is configured to perform attention calculation and key-value caching of the large language model through the GPU, perform routing calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes information of the activated routing experts.

[0013] The request module is used to construct a routing expert execution request based on the routing selection result and the hidden state through the CPU, and submit the routing expert execution request to the FPGA;

[0014] The second execution module is used to perform multilayer perceptron calculations of the activated routing expert through the FPGA according to the routing expert execution request, and obtain a second execution result;

[0015] The fusion module is used to obtain the second execution result through the GPU, and then merge the second execution result with the shared expert output result to continue the subsequent calculation.

[0016] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the implementation method of large language model inference in a heterogeneous system according to any embodiment of the present invention.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the method for implementing large language model inference in a heterogeneous system as described in any embodiment of the present invention.

[0019] The technical solution of this invention can perform attention calculation and key-value caching of a large language model using a GPU, perform routing calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result calculated by the shared expert multilayer perceptron, and the hidden state output by the attention calculation. The routing selection result includes information about the activated routing experts. Then, the CPU constructs a routing expert execution request based on the routing selection result and the hidden state, and submits the routing expert execution request to the FPGA. Finally, the FPGA... Based on the routing expert execution request, the activated routing expert performs multilayer perceptron computation to obtain a second execution result. Finally, the GPU obtains the second execution result and merges it with the shared expert output result to continue subsequent calculations. This allows the FPGA to execute the multilayer perceptron computation of the routing expert. It is evident that this solution achieves large language model inference through the collaboration of CPU, GPU, and FPGA, effectively improving resource utilization. At the same time, the GPU still maintains the execution of the core semantics of the model, such as attention computation, key-value caching, routing computation, and shared expert MLP, avoiding the excessive complexity and risk of model semantic changes brought about by migrating the complete model to the FPGA. Furthermore, by controlling the construction and submission of routing expert execution requests by the CPU, the FPGA does not need to understand the high-level semantics such as scheduling objects, model runner objects, or module structures within the GPU inference framework. It only needs to receive and execute tasks according to a unified request format, thereby achieving decoupling between the FPGA execution domain and the internal implementation of the inference framework, which is beneficial for upgrading or replacing the large language model inference framework.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an implementation method for large language model inference in a heterogeneous system according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of an implementation method for large language model inference in a heterogeneous system according to Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an apparatus for implementing a large language model reasoning method in a heterogeneous system according to Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the method for reasoning with a large language model in a heterogeneous system according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] This embodiment provides a method for implementing large language model inference in a heterogeneous system. The heterogeneous system may include CPU, GPU and FPGA. A large language model is deployed on the CPU, and the large language model may contain at least one MoE layer.

[0030] Figure 1 This is a flowchart illustrating an implementation method for large language model inference in a heterogeneous system, as provided in Embodiment 1 of the present invention. Figure 1 As shown, the method includes:

[0031] S101. The attention calculation and key-value caching of the large language model are performed by the GPU, and the routing calculation and shared expert multilayer perceptron calculation are performed at the MoE layer to obtain the first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes the information of the activated routing experts.

[0032] It should be noted that the reasoning process of a large language model can be a series of reasoning calculations that need to be performed through various network layers in the model after receiving the reasoning task. The calculation process of the MoE layer can be as follows: first, the model's attention calculation and key-value caching are performed, and then the shared expert MLP calculation is performed. Next, the routing calculation is performed to determine the routing experts that need to be activated, and the corresponding routing expert multilayer perceptron calculation is activated. Finally, the output of the routing experts can be fused with the output of the shared experts and then fed into the subsequent layers.

[0033] In this embodiment, in order to reduce the computational resource consumption of the GPU, the GPU can only undertake the responsibility of the backbone computation domain of the large language model inference process, focusing on attention calculation, key-value caching, routing calculation and shared expert multilayer perceptron calculation to obtain the corresponding first execution result, without undertaking the routing expert multilayer perceptron calculation in the MoE layer.

[0034] Attention computation can be understood as the computation process of the self-attention mechanism in a large language model, which is used to capture the dependencies between positions in the input sequence and output the hidden state containing contextual semantic information.

[0035] Key-value cache (KV cache) can be understood as storing and reusing the key tensors and value tensors generated by historical inference steps to avoid repeated calculations of historical input sequences.

[0036] A hybrid expert layer can include multiple routing experts and at least one shared expert. It can dynamically select the most relevant routing experts from the multiple routing expert networks to participate in the computation. For example, each MOE layer can contain one shared expert to handle general knowledge and 256 routing experts to handle specific tasks. During inference, for each input sequence, only the most relevant experts among the 256 routing experts can be activated, eliminating the need for full model computation and reducing computational overhead.

[0037] Building upon this, routing computation can be understood as a decision-making process that calculates which routing experts need to be activated for each input sequence, and can output a routing selection result containing information about the routing experts that need to be activated. Shared expert multilayer perceptron computation can be a common neural network computation that needs to be performed on all input sequences to process general knowledge during the inference process. The shared expert output can be the computation result obtained after performing shared expert multilayer perceptron computation on the input sequences.

[0038] For example, a GPU can load a large language model and perform inference. When inference reaches the MoE layer in the large language model, the GPU can first perform attention calculations prior to the MoE layer to obtain hidden states containing contextual semantic information and store historical key-value pairs in a key-value cache. Simultaneously, the GPU can also first perform shared expert MLP calculations in this MoE layer to obtain the shared expert output results. Then, the GPU can perform routing calculations in the MoE layer to determine the information of the routing experts that need to be activated for each input sequence, obtaining the corresponding routing selection results.

[0039] S102. The CPU constructs a routing expert execution request based on the routing selection result and hidden state, and submits the routing expert execution request to the FPGA.

[0040] It should be noted that, in order to reduce the computational resource consumption of the GPU, this embodiment can use the FPGA in the heterogeneous system to perform the relevant calculations for the routing expert. The CPU can then assume the responsibility of the inference framework control domain, handling the control and coordination of task allocation and data transmission between the GPU and FPGA during the large language model inference process.

[0041] The routing expert execution request can be used to request the FPGA to perform MLP computations for the routing expert that needs to be activated. The routing expert execution request can at least include the routing selection results and hidden states determined by the GPU, or information indicating the storage location of the routing selection results and hidden states, so that the FPGA can determine the routing expert that needs to be activated and read the input data required to perform the multilayer perceptron computation for the routing expert.

[0042] For example, after the GPU completes the routing calculation and attention calculation and obtains the routing selection result and hidden state, the CPU can obtain the storage address information of the routing selection result and hidden state in the GPU video memory, and construct the routing expert execution request based on the storage address information.

[0043] After constructing the routing expert execution request, the CPU can also submit the routing expert execution request to the FPGA's task queue through a pre-established submission queue.

[0044] It should be further explained that although this embodiment aims to reduce GPU computational resource consumption by having the FPGA perform the routing expert calculations, specifically, this embodiment requires the FPGA to only perform the routing expert calculations and not to handle other inference processes such as routing calculations and shared expert multilayer perceptron calculations. The purpose is to preserve the core inference framework of the native model in the GPU, avoiding the complexity and high risk of modifying model semantics that would result from migrating the entire inference framework to the FPGA. Furthermore, in practical engineering, the routing expert calculations (i.e., the routing expert's multilayer perceptron calculations) have a large number of parameters, consuming a significant amount of GPU computation. In contrast, other calculations such as routing calculations and shared expert multilayer perceptron calculations have smaller parameters, and direct execution by the GPU does not consume excessive computational resources, while achieving fast execution and reduced latency.

[0045] Furthermore, by controlling the routing expert through the CPU to construct and submit requests, the FPGA does not need to understand the high-level semantics of the scheduling objects, model runner objects, or module structures within the GPU inference framework. It only needs to receive and execute tasks according to a unified request format, thereby decoupling the FPGA execution domain from the internal implementation of the inference framework, which is beneficial for upgrading or replacing large language model inference frameworks.

[0046] S103. The multilayer perceptron calculation of the activated routing expert is executed by the FPGA according to the routing expert execution request to obtain the second execution result.

[0047] In the heterogeneous computing architecture of this embodiment, the FPGA can assume the role of an external routing expert execution domain, used to execute the MLP calculation tasks of the routing experts activated in the MoE layer.

[0048] The multilayer perceptron (MLP) calculation for the activated routing experts can be performed on the hidden state based on the information of the activated routing experts in the routing selection results. The second execution result can be understood as the comprehensive result after summarizing the MLP calculations of all activated routing experts.

[0049] For example, after receiving the routing expert execution request submitted by the CPU, the FPGA can first parse the routing expert execution request, determine the corresponding routing selection result and hidden state, then determine the corresponding activated routing experts according to the routing selection result, and perform multilayer perceptron calculation on the hidden state by calling each activated routing expert, and finally summarize all the calculation results to obtain the second execution result.

[0050] S104. Obtain the second execution result through the GPU, and then merge the second execution result with the shared expert output result to continue the subsequent calculation.

[0051] The GPU can obtain the second execution result output by the FPGA in two ways: either the FPGA writes the second execution result to a preset storage location in the GPU, or the GPU directly reads the second execution result stored in the preset location in the FPGA.

[0052] After obtaining the second execution result, the GPU can fuse the second execution result with the shared expert output result to obtain the final output of the current MoE layer, and continue to execute the subsequent neural network operations in the large language model based on the final output until the complete inference process of the current input sequence is completed.

[0053] The technical solution of this invention can perform attention calculation and key-value caching of a large language model using a GPU, perform routing calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result calculated by the shared expert multilayer perceptron, and the hidden state output by the attention calculation. The routing selection result includes information about the activated routing experts. Then, the CPU constructs a routing expert execution request based on the routing selection result and the hidden state, and submits the routing expert execution request to the FPGA. Finally, the FPGA... Based on the routing expert execution request, the activated routing expert performs multilayer perceptron computation to obtain a second execution result. Finally, the GPU obtains the second execution result and merges it with the shared expert output result to continue subsequent calculations. This allows the FPGA to execute the multilayer perceptron computation of the routing expert. By coordinating the CPU, GPU, and FPGA to achieve large language model inference, resource utilization is effectively improved. At the same time, the GPU still maintains the execution of the core semantics of the model, such as attention computation, key-value caching, routing computation, and shared expert MLP, avoiding the excessive complexity and risk of model semantic changes brought about by migrating the complete model to the FPGA. Furthermore, by controlling the construction and submission of routing expert execution requests by the CPU, the FPGA does not need to understand the high-level semantics such as scheduling objects, model runner objects, or module structures inside the GPU inference framework. It only needs to receive and execute tasks according to a unified request format, thereby decoupling the FPGA execution domain from the internal implementation of the inference framework, which is beneficial for upgrading or replacing the large language model inference framework.

[0054] Based on the above embodiments, the step of constructing a routing expert execution request by the CPU based on the routing selection result and the hidden state may include:

[0055] The CPU constructs a routing expert to execute the request based on the routing results and hidden states, according to the predefined Application Binary Interface (ABI) specification.

[0056] It should be noted that, in order to standardize data exchange and collaborative work between GPUs and FPGAs, ABI specifications can be predefined so that all parties can describe data and construct requests according to a unified standard.

[0057] The ABI specification can be a pre-agreed device protocol boundary between the CPU control domain and the FPGA external expert execution domain. It is used to define the data structure format, field meaning and encoding method of routing expert execution requests, so that the FPGA can parse and execute routing expert execution requests regardless of the inference framework version.

[0058] The advantage of this setup is that by constructing routing expert execution requests through a predefined Application Binary Interface (ABI) specification, the access of the FPGA's expert execution domain can be uniformly adapted to the MoE layer of different versions of the inference framework. When the inference framework version upgrade causes changes in internal objects, MoE module structures, or scheduling objects, the FPGA's external execution domain still receives requests and returns responses through a stable ABI, without needing to understand the internal objects of the new version of the inference framework.

[0059] Example 2

[0060] Figure 2 This is a flowchart illustrating a method for implementing large language model inference in a heterogeneous system, as provided in Embodiment 2 of the present invention. This embodiment can be further improved and described based on the above embodiments. Figure 2 As shown, the method includes:

[0061] S201. The attention calculation and key-value caching of the large language model are performed by the GPU, and the routing calculation and shared expert multilayer perceptron calculation are performed at the MoE layer to obtain the first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes the information of the activated routing experts.

[0062] S202. Create input tensor descriptors and output tensor descriptors conforming to the predefined ABI specification through the CPU; the input tensor descriptor is used to indicate the storage location and data format of the routing results and hidden state in the GPU; the output tensor descriptor is used to indicate the storage location and data format of the second execution result in the GPU.

[0063] It's important to note that a tensor descriptor (TensorDesc) is a data structure defined in the ABI specification, used to describe the tensor boundary information required for cross-device execution. A tensor descriptor can include the type and identifier of the device containing the tensor, the tensor's storage address, data precision type, tensor shape (number of dimensions and size of each dimension) or step size (interval between adjacent elements in each dimension), etc. The tensor descriptor only expresses the tensor boundary information required for cross-device execution and does not contain tensor objects or Python objects within the inference framework. Furthermore, the input tensor descriptor can be an ABI-compliant data structure describing the storage location and data format of routing results and hidden states in the GPU. The output tensor descriptor can be an ABI-compliant data structure describing the storage location and data format of the second execution result in the GPU.

[0064] For example, after the GPU completes the route calculation and obtains the route selection result, the CPU can obtain the storage address information of the route selection result and the hidden state in the GPU video memory. Then, based on the storage address information, it generates an input tensor descriptor according to a preset data structure format. At the same time, it obtains the GPU storage address information pre-allocated for the calculation result of the routing expert MLP. Then, based on the storage address information, it generates an output tensor descriptor according to a preset data structure format.

[0065] S203. Create a route descriptor conforming to the ABI specification via the CPU; the route descriptor is used to indicate the storage location of the expert model parameters of the activated route expert on the FPGA.

[0066] A route descriptor (RouteDesc) is a data structure conforming to the ABI specification that describes the storage location of expert model parameters for activated route experts within the FPGA. The route descriptor can include at least the storage address of the expert weights corresponding to the activated route expert identifiers in the FPGA, the number of valid route experts (active_top_k), the order of the outputs of each route expert in the result tensor (i.e., expert slot order information), and the route semantic version. Through the route descriptor, the FPGA can correctly locate and load the corresponding route expert based on the GPU's routing computation results.

[0067] For example, the CPU can determine the address information of the expert model parameters of the corresponding routing expert in the FPGA based on the routing selection result, and generate a routing descriptor according to a preset data structure format.

[0068] S204. The CPU creates a routing expert execution request conforming to the ABI specification based on the input tensor descriptor, output tensor descriptor, and routing descriptor, and submits the routing expert execution request to the FPGA.

[0069] Understandably, the routing expert execution request may include an input tensor descriptor indicating the storage location of the routing selection results and hidden states in the GPU, so that the FPGA can read from the GPU the input data required to execute the routing expert multilayer perceptron computation, the routing descriptor indicating the storage location of the expert model parameters of the routing expert to be activated in the FPGA, so that the FPGA can correctly load the corresponding routing expert, and an output tensor descriptor indicating the target storage location of the routing expert multilayer perceptron MLP computation results in the GPU, so that the FPGA can write the computation results to the GPU for subsequent use.

[0070] The advantage of this setup is that by constructing input tensor descriptors, output tensor descriptors, and routing descriptors to create routing expert execution requests, the source of input data required for FPGA computation, the destination of output data, and the location of expert parameters to be called can be clearly defined. This enables the FPGA to autonomously complete data reading, computation execution, and result writing back according to the ABI specification, reducing its dependence on objects within the inference framework.

[0071] S205. Using the FPGA based on the input tensor descriptor, read the activated routing expert identifier, the route weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool.

[0072] It should be noted that the information of the activated routing experts can include: the activated routing expert identifier (topk_ids) and the activated routing expert's routing weight (topk_weights). The activated routing expert identifier can be a number or index used to uniquely identify the routing expert to be activated. Each routing expert corresponds to one routing expert identifier. The routing weight of the activated routing expert can be used to characterize the proportion of each activated routing expert's output during fusion.

[0073] The GPU buffer pool can be a buffer storage location pre-established in the GPU's video memory. The buffer pool can include an input buffer for storing the input data required by the FPGA and an output buffer for storing the FPGA's output data. The input buffer can store the activated routing expert identifiers, corresponding routing weights, and hidden states of the attention calculation output generated by the GPU routing computation. The output buffer can be used to store the second execution results output by the FPGA.

[0074] Understandably, since the buffer pool is pre-established, there is no need to repeatedly register or unregister the GPU buffer for each MOE layer during the execution phase, which reduces the fixed overhead of each MOE layer call.

[0075] For example, after receiving a routing expert execution request, the FPGA can first read the activated routing expert identifier, the routing weight corresponding to each routing expert, and the hidden state data from the input buffer of the GPU buffer pool according to the storage location indicated in the input tensor descriptor.

[0076] S206. Using the FPGA, the expert model parameters of the activated routing expert are found based on the activated routing expert identifier and routing descriptor. The multilayer perceptron calculation of the activated routing expert is then performed using the expert model parameters. Based on the routing weight of the activated routing expert, the sub-results obtained from the multilayer perceptron calculation of each activated routing expert are weighted and fused to obtain the second execution result.

[0077] The expert model parameters can be the parameters used by the routing experts to perform multilayer perceptron calculations. Sub-results can be the intermediate outputs obtained by each activated routing expert performing multilayer perceptron calculations individually.

[0078] For example, the FPGA can look up the expert model parameters of the corresponding routing expert in the storage location indicated by the routing descriptor based on the activated routing expert's identifier. For each activated routing expert, the FPGA can use the found expert model parameters to perform multilayer perceptron computation on the hidden state to obtain sub-results for each routing expert. Then, the FPGA can perform weighted summation or other weighted fusion operations on the sub-results based on the routing weights of each routing expert to obtain the final second execution result.

[0079] S207. The second execution result is written to the output buffer in the GPU buffer pool through the FPGA based on the output tensor descriptor.

[0080] For example, the FPGA can write the second execution result into the output buffer in the GPU buffer pool according to the storage location indicated in the output tensor descriptor, so that the GPU can read it later.

[0081] The advantage of this setup is that the pre-established GPU buffer pool allows the FPGA to directly read input data from the GPU buffer pool and write output data, avoiding the overhead of buffer registration and deregistration during each layer call and reducing inference latency.

[0082] S208: Read the second execution result through the GPU, and merge the second execution result with the shared expert output result before continuing to execute subsequent calculations.

[0083] The technical solution of this embodiment uses a GPU to perform attention calculation and key-value caching of a large language model, and performs routing calculation and shared expert multilayer perceptron calculation at the MoE layer to obtain a first execution result. Then, the CPU can create input tensor descriptors, output tensor descriptors, and routing descriptors that conform to the predefined ABI specification, thereby creating a routing expert execution request that conforms to the ABI specification. Next, the FPGA can read the activated routing expert identifier, the routing weight, and the hidden state from the input buffer in the pre-established GPU buffer pool based on the input tensor descriptor. Then, based on the activated routing expert identifier and routing descriptor, the expert model parameters of the activated routing expert are found, and the multilayer perceptron calculation of the activated routing expert is performed using the expert model parameters. Based on the routing weight of the activated routing expert, the sub-results obtained from the multilayer perceptron calculation of each activated routing expert are weighted and fused to obtain a second execution result. The second execution result is written to the output buffer in the GPU buffer pool. Finally, the GPU can read the second execution result and fuse it with the shared expert output result in the first execution result to continue subsequent operations. As can be seen, this embodiment can further create routing expert execution requests through input tensor descriptors, output tensor descriptors, and routing descriptors, enabling the FPGA to autonomously complete data reading, computation execution, and result writing back according to the ABI specification, reducing dependence on objects within the inference framework. Furthermore, it also allows the FPGA to directly read input data from the GPU buffer pool and write output data, avoiding the buffer registration and deregistration overhead during each layer call and reducing inference latency.

[0084] Based on the above embodiments, the input tensor descriptor and output tensor descriptor may specifically include at least one of the following: GPU virtual address pointer, registered memory handle, DMA physical bus address, PCIe BAR1 base address, memory space type, device type identifier, GPU unique identifier hash value, PCIe root complex identifier, total data byte length, tensor dimension shape, tensor stride, data precision type, memory alignment byte count, tensor storage layout, buffer pool identifier, buffer slot identifier, lifecycle, data producer identifier, memory access mode, and data checksum; the routing descriptor may specifically include: total number of active experts and expert slot order information.

[0085] The GPU virtual address pointer can be a virtual address visible to the GPU side, used to locate the starting position of tensor data in the GPU memory space. The registered memory handle can be an access handle obtained after the buffer is registered during the ABI session negotiation phase, used for quickly indexing pre-registered buffer information. The DMA physical bus address and PCIe BAR1 base address are the physical addresses used by the FPGA for P2P DMA access, corresponding to the PCIe bus address and BAR1 space base address, respectively. The memory space type indicates whether the storage location is in GPU video memory or CPU host memory, used to determine the access path and address translation method. The device type identifier can be used to identify the type of device (e.g., GPU, FPGA, or CPU), and the GPU unique identifier hash value can be used to uniquely identify GPU devices in a multi-GPU system. The PCIe root complex identifier can be used to identify the PCIe domain to which the device belongs in a multi-root complex system. The total data byte length indicates the total number of bytes occupied by the tensor data, used to control boundary checks and DMA transfer length. Tensor dimension shape indicates the size of each dimension of the tensor, while tensor stride indicates the byte spacing between adjacent elements in each dimension. Shape and stride together describe the tensor's data structure layout. Data precision type indicates the data type of tensor elements (e.g., FP32, FP16, INT8). Memory alignment bytes indicate the alignment requirements of the buffer's starting address, used to satisfy alignment constraints for specific hardware access. Tensor storage layout indicates how tensors are arranged in memory (e.g., row-major or column-major). Buffer pool identifier and buffer slot identifier are used to locate a specific buffer in a pre-registered buffer pool; their combination identifies a unique buffer slot. Lifetime can be a lifetime version identifier for a buffer slot, used to prevent the ABA problem during buffer reuse (i.e., completion notifications of old requests mistakenly hit new requests after reuse). Data producer identifier can identify the producer of tensor data (e.g., GPU routing computation), used for data source tracing and debugging. Memory access mode indicates read / write permissions for the buffer (e.g., read-only, write-only, read / write), used for access permission verification. Data checksums are used for data integrity verification and can be either CRC or hash values.

[0086] The total number of active experts (active_top_k) indicates the number of routing experts actually activated in the current route selection, i.e., the k value taken for top-k in the route calculation. The expert slot order information (topk_slot_order) indicates the order in which the outputs of each routing expert are arranged in the result tensor, enabling the FPGA to correctly map the sub-results of each expert to the route weights and perform weighted fusion.

[0087] Furthermore, as one possible implementation of this embodiment, the FPGA reads the activated routing expert's identifier, routing weight, and hidden state from the pre-established GPU buffer pool based on the input tensor descriptor. Specifically, this may include:

[0088] By using the FPGA and employing a point-to-point direct memory access (P2P DMA) method based on the input tensor descriptor, the activated routing expert identifier, the route weight of the activated routing expert, and the hidden state are read from the input buffer in the pre-established GPU buffer pool.

[0089] The second execution result is written to the output buffer in the GPU buffer pool via the FPGA based on the output tensor descriptor. Specifically, this can include:

[0090] The second execution result is written to the output buffer in the GPU buffer pool using a P2P DMA method based on the output tensor descriptor via the FPGA.

[0091] Among them, the P2P DMA method performs data transmission based on the device address field contained in the input tensor descriptor. The device address field is registered during the ABI session negotiation phase, which is a negotiation process between the CPU and the FPGA in advance.

[0092] It should be noted that the P2P DMA method can be an access mechanism in which the FPGA directly accesses the GPU memory through the PCIe bus without going through the CPU as an intermediary.

[0093] The device address field can be the mapping address of the input buffer or output buffer in the GPU buffer pool in the device address space.

[0094] For example, in this embodiment, during the ABI session negotiation phase, the device address fields of each buffer in the GPU buffer pool can be negotiated and registered in advance between the CPU and the FPGA to establish a direct access channel between the FPGA and the GPU memory.

[0095] During the routing expert calculation phase, the FPGA can directly read data from the GPU buffer pool's input buffer via P2P DMA, based on the device address field contained in the input tensor descriptor. After the routing expert calculation is complete, the FPGA can directly write the second execution result into the GPU buffer pool's output buffer via P2P DMA, based on the device address field contained in the output tensor descriptor.

[0096] The advantages of this setup are as follows: Through P2P DMA, direct data transfer between the FPGA and GPU avoids the latency issues caused by data being transferred through the CPU memory during routing calculations, thus improving data transfer efficiency. Furthermore, by negotiating and registering the device address fields of each buffer in the GPU buffer pool in advance during the ABI session negotiation phase, the FPGA can obtain the target address of the P2P DMA method simply by parsing the routing descriptor, without needing to understand the details of GPU memory management, further improving data transfer efficiency.

[0097] Based on the above embodiments, the present invention also provides an optional embodiment that can further optimize the method for creating a routing expert to execute a request:

[0098] The CPU creates a routing expert execution request conforming to the ABI specification based on the input tensor descriptor, output tensor descriptor, and route descriptor, which may include:

[0099] A synchronization descriptor conforming to the ABI specification is created through the CPU; the synchronization descriptor is used to indicate synchronization flags that keep data read and write operations synchronized between the GPU and FPGA.

[0100] The CPU creates ABI-compliant routing expert execution requests based on input tensor descriptors, output tensor descriptors, routing descriptors, and synchronization descriptors.

[0101] The synchronization descriptor (SyncDesc) can be an ABI-compliant data structure used to control the timing of data exchange between the GPU and FPGA. It ensures that the FPGA reads the required input data after the GPU data is ready, and the GPU reads the FPGA output data after the FPGA has finished writing. Synchronization flags can be identification information used to characterize the data ready state or the operation completion state, such as input ready flags, completion flags, event flags, or fence signals.

[0102] For example, after creating the input tensor descriptor, output tensor descriptor, and routing descriptor, the CPU can further create a synchronization descriptor to indicate that the GPU and FPGA are ready for data. Then, the CPU can combine the synchronization descriptor, tensor descriptor, output tensor descriptor, and routing descriptor to generate a routing expert execution request.

[0103] Specifically, a synchronization descriptor may include: an input ready flag;

[0104] Before reading the activated route expert identifier, the route weight of the activated route expert, and the hidden state from the input buffer in the pre-established GPU buffer pool via the FPGA based on the input tensor descriptor, the method may further include:

[0105] The FPGA verifies whether the first execution result is ready in the input buffer of the GPU buffer pool based on the input ready flag. If so, it triggers the execution of the operation based on the input tensor descriptor, which reads the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer of the pre-established GPU buffer pool.

[0106] The input ready flag (input_ready_ref) can be a flag in the synchronization descriptor used to indicate whether the GPU has completed the write operation of relevant data in the first execution result to the input buffer. It can be used to indicate whether the GPU has completed the write operation of relevant data in the first execution result (including the activated route expert identifier, the corresponding route weight, and the hidden state of the attention calculation output) to the input buffer.

[0107] For example, after receiving a routing expert execution request, the FPGA can first check the input ready flag in the synchronization descriptor, which can be set by the GPU after completing the writing of the first execution result. If the input ready flag indicates that the data is ready (e.g., the flag value is a preset value or the corresponding event has been triggered), the FPGA can start executing the data reading operation; if the input ready flag indicates that the data is not yet ready, the FPGA enters a waiting state or returns to a retry state.

[0108] The advantage of this setup is that it ensures the FPGA only starts reading and executing data after the input data is fully ready, avoiding the reading of incomplete or unread data, thus improving the reliability and data consistency of the inference process.

[0109] In addition, a synchronization descriptor may also include: a completion flag;

[0110] Obtaining a second execution result via the GPU can include:

[0111] The completion flag in the routing expert execution response returned by the GPU based on the FPGA verifies whether the FPGA has successfully written the second execution result to the output buffer in the GPU buffer pool. If so, the second execution result is read. The routing expert execution response is the response message returned by the FPGA after obtaining the second execution result.

[0112] The completion flag (completion_fence) can be a flag in the synchronization descriptor used to indicate whether the FPGA has completed the second execution result write operation to the output buffer. It can be included in the routing expert execution response returned by the FPGA. This routing expert execution response can be a status message returned by the FPGA to the GPU after completing the routing expert multilayer perceptron calculation and writing the result back to the GPU. In addition to the completion flag, the routing expert execution response may also include a request identifier, layer identifier, execution status code, and error code.

[0113] For example, after completing the calculation and write-back of the second execution result, the FPGA can generate and return a routing expert execution response, setting the completion flag in it to a valid state.

[0114] Before reading the second execution result, the GPU can check the completion flag in the routing expert's execution response. If the completion flag indicates that the FPGA has successfully written the second execution result to the output buffer, the GPU can read the second execution result from the output buffer; if the completion flag indicates that the write was unsuccessful or has not yet been completed, the GPU will trigger error handling or wait.

[0115] The advantage of this setting is that it prevents the GPU from reading the routing expert's calculation results before they are fully written.

[0116] Based on any of the above implementation methods, the method may also include:

[0117] The FPGA performs ABI verification and / or information consistency verification on the received routing expert execution request. If the verification fails, the routing expert execution request is rejected.

[0118] Among them, ABI verification is used to verify whether the routing expert execution request conforms to the ABI specification, and consistency verification is used to verify whether the information in the routing expert execution request is consistent with the result of ABI session negotiation. ABI session negotiation is a negotiation process between the CPU and the FPGA in advance.

[0119] ABI verification can involve the FPGA checking whether the format, field integrity, data type matching, and version number consistency of the routing expert's execution request conform to the ABI specification definition. Information consistency verification can involve the FPGA verifying whether information such as data type, tensor layout, synchronization method, and device address field in the request is consistent with the capability parameters and registration information determined during the ABI session negotiation phase.

[0120] For example, after receiving a routing expert's execution request, the FPGA can check whether the message format of the request conforms to the ABI specification, including whether the fields of each descriptor are complete and whether the ABI version numbers are consistent. Simultaneously, the FPGA verifies whether the data type in the request is a type declared as supported during the negotiation phase, whether the tensor layout is consistent with the negotiation result, and whether the device address field belongs to the registered valid address range.

[0121] If any check item fails, the FPGA can refuse to execute the routing expert execution request, return an error response containing an error code to the CPU, and release the allocated computing resources.

[0122] The advantage of this setup is that by performing ABI verification and information consistency verification on the FPGA side, protocol mismatches or parameter errors can be detected in time before computation is executed, preventing the FPGA from performing invalid computations or accessing illegal memory addresses based on erroneous requests, thus improving the security of inference. At the same time, it ensures protocol consistency between the external expert execution domain and the inference framework control domain.

[0123] Example 3

[0124] Figure 3 This is a schematic diagram of the structure of a device for implementing large language model reasoning in a heterogeneous system, provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0125] The first execution module 301 can be used to perform attention calculation and key-value caching of a large language model through the GPU, perform route calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the route selection result determined by the route calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The route selection result includes the information of the activated route experts.

[0126] Request module 302 can be used to construct a routing expert execution request based on the routing selection result and hidden state through the CPU, and submit the routing expert execution request to the FPGA;

[0127] The second execution module 303 can be used to execute the multilayer perceptron calculation of the activated routing expert through the FPGA according to the routing expert execution request, and obtain the second execution result;

[0128] The fusion module 304 can be used to obtain the second execution result through the GPU, and then merge the second execution result with the shared expert output result to continue the subsequent calculation.

[0129] The technical solution of this embodiment can perform attention calculation and key-value caching of a large language model using a GPU, perform routing calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result from the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes information about the activated routing experts. Then, the CPU constructs a routing expert execution request based on the routing selection result and the hidden state, and submits the routing expert execution request to the FPGA. Finally, the FPGA... Based on the routing expert execution request, the activated routing expert performs multilayer perceptron computation to obtain a second execution result. Finally, the GPU obtains the second execution result and merges it with the shared expert output result to continue subsequent calculations. This allows the FPGA to execute the multilayer perceptron computation of the routing expert. By coordinating the CPU, GPU, and FPGA to achieve large language model inference, resource utilization is effectively improved. At the same time, the GPU still maintains the execution of the core semantics of the model, such as attention computation, key-value caching, routing computation, and shared expert MLP, avoiding the excessive complexity and risk of model semantic changes brought about by migrating the complete model to the FPGA. Furthermore, by controlling the construction and submission of routing expert execution requests by the CPU, the FPGA does not need to understand the high-level semantics such as scheduling objects, model runner objects, or module structures inside the GPU inference framework. It only needs to receive and execute tasks according to a unified request format, thereby decoupling the FPGA execution domain from the internal implementation of the inference framework, which is beneficial for upgrading or replacing the large language model inference framework.

[0130] Optionally, the request module 302 can be specifically used to construct a route expert to execute a request based on the routing results and the hidden state, according to the predefined Application Binary Interface (ABI) specification, through the CPU.

[0131] Optionally, request module 302 may include:

[0132] The first building unit can be used to create input tensor descriptors and output tensor descriptors that conform to a predefined ABI specification via the CPU; the input tensor descriptor is used to indicate the storage location and data format of the routing results and hidden state in the GPU; the output tensor descriptor is used to indicate the storage location and data format of the second execution result in the GPU.

[0133] The second building unit can be used to create a route descriptor that conforms to the ABI specification via the CPU; the route descriptor is used to indicate the storage location of the expert model parameters of the activated route expert on the FPGA.

[0134] The creation unit can be used to create routing expert execution requests that conform to the ABI specification by using the CPU based on the input tensor descriptor, output tensor descriptor, and routing descriptor.

[0135] Optionally, the information of the activated routing expert may include: the activated routing expert's identifier and the activated routing expert's routing weight.

[0136] Correspondingly, the second execution module 303 may include:

[0137] The reading unit can be used to read the activated routing expert's identifier, routing weight, and hidden state from the input buffer in the pre-established GPU buffer pool via the FPGA based on the input tensor descriptor.

[0138] The execution unit can be used to find the expert model parameters of the activated routing expert based on the activated routing expert identifier and routing descriptor through the FPGA, and use the expert model parameters to perform multilayer perceptron calculation of the activated routing expert. Based on the routing weight of the activated routing expert, the sub-results obtained from the multilayer perceptron calculation of each activated routing expert are weighted and fused to obtain the second execution result.

[0139] The write unit can be used to write the second execution result into the output buffer in the GPU buffer pool via the FPGA based on the output tensor descriptor.

[0140] Optionally, the read unit can be used to read the activated routing expert identifier, the route weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool via the FPGA using point-to-point direct memory access (P2P DMA) based on the input tensor descriptor.

[0141] The write unit can be used to write the second execution result into the output buffer in the GPU buffer pool via the FPGA using the P2P DMA method based on the output tensor descriptor.

[0142] Among them, the P2P DMA method performs data transmission based on the device address field contained in the input tensor descriptor. The device address field is registered during the ABI session negotiation phase, which is a negotiation process between the CPU and the FPGA in advance.

[0143] Optionally, request module 302 may also include:

[0144] The third building block can be used to create synchronization descriptors that conform to the ABI specification via the CPU; the synchronization descriptor is used to indicate synchronization flags that keep data read and write operations between the GPU and FPGA synchronized.

[0145] Correspondingly, the creation unit can be used to create a routing expert execution request that conforms to the ABI specification by using the CPU based on the input tensor descriptor, output tensor descriptor, routing descriptor, and synchronization descriptor.

[0146] Optionally, the synchronization descriptor may include: an input ready flag;

[0147] Correspondingly, the second execution module 303 may also include:

[0148] The first verification unit can be used to verify, based on the input ready flag, whether the first execution result is ready in the input buffer of the GPU buffer pool. If so, it triggers the execution of the operation of reading the activated routing expert identifier, the routing weight of the activated routing expert and the hidden state from the input buffer of the pre-established GPU buffer pool based on the input tensor descriptor.

[0149] Optionally, the synchronization descriptor may also include: a completion flag;

[0150] Correspondingly, the fusion module 304 can be used to verify whether the FPGA has successfully written the second execution result into the output buffer in the GPU buffer pool by using the completion flag in the routing expert execution response returned by the GPU based on the FPGA. If so, the second execution result is read. The routing expert execution response is the response message returned by the FPGA after obtaining the second execution result.

[0151] Optionally, the input tensor descriptor and the output tensor descriptor may include at least one of the following: GPU virtual address pointer, registered memory handle, DMA physical bus address, PCIe BAR1 base address, memory space type, device type identifier, GPU unique identifier hash value, PCIe root complex identifier, total data byte length, tensor dimension shape, tensor stride, data precision type, memory alignment bytes, tensor storage layout, buffer pool identifier, buffer slot identifier, lifecycle, data producer identifier, memory access mode, and data checksum;

[0152] The route descriptor may include: the total number of active experts and the expert slot order information.

[0153] Optionally, the device may further include:

[0154] The verification module can be used to perform ABI verification and / or information consistency verification on the received routing expert execution request via FPGA. If the verification fails, the routing expert execution request will be rejected.

[0155] Among them, ABI verification is used to verify whether the routing expert execution request conforms to the ABI specification, and consistency verification is used to verify whether the information in the routing expert execution request is consistent with the result of ABI session negotiation. ABI session negotiation is a negotiation process between the CPU and the FPGA in advance.

[0156] Example 4

[0157] Figure 4 A schematic diagram of an electronic device 400 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0158] like Figure 4 As shown, the electronic device 400 includes at least one processor 401 and a memory, such as a read-only memory (ROM) 402 or a random access memory (RAM) 403, communicatively connected to the at least one processor 401. The memory stores computer programs executable by the at least one processor. The processor 401 can perform various appropriate actions and processes based on the computer program stored in the ROM 402 or loaded into the RAM 403 from storage unit 408. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0159] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0160] Processor 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 401 performs the various methods and processes described above, such as implementations of large language model inference in heterogeneous systems.

[0161] In some embodiments, the implementation method of large language model inference in heterogeneous systems can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the implementation method of large language model inference in heterogeneous systems described above can be performed. Alternatively, in other embodiments, processor 401 can be configured to execute the implementation method of large language model inference in heterogeneous systems by any other suitable means (e.g., by means of firmware).

[0162] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0167] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0168] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the method for implementing large language model inference in a heterogeneous system according to any embodiment of the present invention.

[0169] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for implementing large language model reasoning in a heterogeneous system, characterized in that, The heterogeneous system includes a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). A large language model is deployed on the CPU, and the large language model includes at least one hybrid expert MoE layer. The method includes: The GPU performs attention calculation and key-value caching of the large language model, performs routing calculation at the MoE layer, and performs shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes information about the activated routing experts. The CPU constructs a routing expert execution request based on the routing selection result and the hidden state, and submits the routing expert execution request to the FPGA; The second execution result is obtained by performing multilayer perceptron calculations on the activated routing expert through the FPGA according to the routing expert execution request; The second execution result is obtained through the GPU, and the second execution result is fused with the shared expert output result before continuing to execute subsequent operations.

2. The method according to claim 1, characterized in that, The step of constructing a routing expert execution request based on the routing selection result and the hidden state using the CPU includes: The CPU constructs a routing expert execution request based on the routing results and the hidden state, according to the predefined Application Binary Interface (ABI) specification.

3. The method according to claim 2, characterized in that, The step of constructing a routing expert execution request based on the routing selection result and the hidden state by the CPU according to the predefined ABI specification includes: The CPU creates input and output tensor descriptors that conform to a predefined ABI specification; the input tensor descriptor indicates the storage location and data format of the routing result and the hidden state on the GPU; the output tensor descriptor indicates the storage location and data format of the second execution result on the GPU. The CPU creates a route descriptor that conforms to the ABI specification; the route descriptor is used to indicate the storage location of the expert model parameters of the activated route expert in the FPGA. The CPU creates a routing expert execution request that conforms to the ABI specification based on the input tensor descriptor, the output tensor descriptor, and the routing descriptor.

4. The method according to claim 3, characterized in that, The information of the activated routing expert includes: the identifier of the activated routing expert and the routing weight of the activated routing expert. The step of performing multilayer perceptron calculations on the activated routing expert via the FPGA according to the routing expert execution request to obtain a second execution result includes: The FPGA reads the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool based on the input tensor descriptor. The FPGA searches for the expert model parameters of the activated routing expert based on the activated routing expert identifier and the routing descriptor, and uses the expert model parameters to perform multilayer perceptron calculations for the activated routing expert. Based on the routing weights of the activated routing experts, the sub-results obtained from the multilayer perceptron calculations of each activated routing expert are weighted and fused to obtain the second execution result. The second execution result is written into the output buffer in the GPU buffer pool by the FPGA based on the output tensor descriptor.

5. The method according to claim 4, characterized in that, The step of reading the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool based on the input tensor descriptor via the FPGA includes: The FPGA uses a point-to-point direct memory access (P2P DMA) method based on the input tensor descriptor to read the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool. The step of writing the second execution result into the output buffer in the GPU buffer pool via the FPGA based on the output tensor descriptor includes: The second execution result is written into the output buffer of the GPU buffer pool using the P2P DMA method based on the output tensor descriptor via the FPGA. The P2P DMA method performs data transmission based on the device address field contained in the input tensor descriptor. The device address field is registered during the ABI session negotiation phase, which is a negotiation process between the CPU and the FPGA in advance.

6. The method according to claim 4, characterized in that, The step of creating a routing expert execution request conforming to the ABI specification by means of the CPU based on the input tensor descriptor, the output tensor descriptor, and the routing descriptor includes: The CPU creates a synchronization descriptor that conforms to the ABI specification; the synchronization descriptor is used to indicate a synchronization flag that keeps data read and write operations between the GPU and the FPGA synchronized. The CPU creates a routing expert execution request conforming to the ABI specification based on the input tensor descriptor, the output tensor descriptor, the routing descriptor, and the synchronization descriptor.

7. The method according to claim 6, characterized in that, The synchronization descriptor includes: an input ready flag; Before reading the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer in the pre-established GPU buffer pool via the FPGA based on the input tensor descriptor, the method further includes: The FPGA verifies whether the first execution result is ready in the input buffer of the GPU buffer pool based on the input ready flag. If so, it triggers the operation of reading the activated routing expert identifier, the routing weight of the activated routing expert, and the hidden state from the input buffer of the pre-established GPU buffer pool based on the input tensor descriptor.

8. The method according to claim 6, characterized in that, The synchronization descriptor includes: a completion flag; The step of obtaining the second execution result through the GPU includes: The GPU verifies whether it has successfully written the second execution result into the output buffer of the GPU buffer pool by using the completion flag in the routing expert execution response returned by the FPGA. If so, the second execution result is read. The routing expert execution response is a response message returned by the FPGA after obtaining the second execution result.

9. The method according to claim 3, characterized in that, The input tensor descriptor and the output tensor descriptor include at least one of the following: GPU virtual address pointer, registered memory handle, DMA physical bus address, PCIe BAR1 base address, memory space type, device type identifier, GPU unique identifier hash value, PCIe root complex identifier, total data byte length, tensor dimension shape, tensor stride, data precision type, memory alignment byte count, tensor storage layout, buffer pool identifier, buffer slot identifier, lifecycle, data producer identifier, memory access mode, and data checksum; The routing descriptor includes: the total number of active experts and the order of expert slots.

10. The method according to any one of claims 2-9, characterized in that, The method further includes: The FPGA performs ABI verification and / or information consistency verification on the received routing expert execution request. If the verification fails, the routing expert execution request is rejected. The ABI verification is used to verify whether the routing expert execution request conforms to the ABI specification, and the consistency verification is used to verify whether the information in the routing expert execution request is consistent with the result of the ABI session negotiation. The ABI session negotiation is a negotiation process between the CPU and the FPGA in advance.

11. A device for implementing large language model reasoning in a heterogeneous system, characterized in that, The heterogeneous system includes a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). A large language model is deployed on the CPU, and the large language model includes at least one hybrid expert MoE layer. The device includes: The first execution module is configured to perform attention calculation and key-value caching of the large language model through the GPU, perform routing calculation at the MoE layer, and perform shared expert multilayer perceptron calculation to obtain a first execution result. The first execution result includes the routing selection result determined by the routing calculation, the shared expert output result of the shared expert multilayer perceptron calculation, and the hidden state output by the attention calculation. The routing selection result includes information of the activated routing experts. The request module is used to construct a routing expert execution request based on the routing selection result and the hidden state through the CPU, and submit the routing expert execution request to the FPGA; The second execution module is used to perform multilayer perceptron calculations of the activated routing expert through the FPGA according to the routing expert execution request, and obtain a second execution result; The fusion module is used to obtain the second execution result through the GPU, and then merge the second execution result with the shared expert output result to continue the subsequent calculation.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform an implementation method for large language model reasoning in a heterogeneous system according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the method for implementing large language model reasoning in a heterogeneous system as described in any one of claims 1-10.

Images