Distributed heterogeneous big language model reasoning system and reasoning method

Abstract

The invention provides a distributed heterogeneous big language model inference system, the system is used for executing a big language model inference task, the system comprises an application interface module, a scheduling module and a plurality of heterogeneous calculation modules which are orderly arranged according to a pipeline mode, the application interface module is configured with a plurality of application interfaces, and the scheduling module is configured with a plurality of heterogeneous calculation modules. Each application interface can be called by all computing modules to adapt to a computing mode implementation strategy of the own structure; the scheduling module is used for constructing a model scheduling scheme according to the structure of a large language model and periodically packaging a plurality of reasoning tasks into task packages; each calculation module is configured to schedule an allocated calculation layer according to a model scheduling scheme, call a calculation mode implementation strategy matched with the structure of the calculation module from the application interface module through a corresponding application interface, and sequentially infer each inference task in the task package in a pipeline parallel mode; and obtaining a reasoning result of each reasoning task.

Description

A Distributed Heterogeneous Large Language Model Inference System and Inference Method Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, specifically to a distributed reasoning technology for large language models, and more specifically to a distributed heterogeneous large language model reasoning system and reasoning method. Background Technology

[0002] In recent years, generative artificial intelligence technology, represented by large language models (LLM), has developed rapidly, and its application scenarios have expanded from academic research to all walks of life. However, the rapid expansion of the scale of LLM models (from billions to trillions of parameters) makes it impossible for them to run on a single computing device. Distributed computing technology must be adopted to divide the model into multiple or even thousands of accelerator cards for collaborative operation.

[0003] Current mainstream LLM inference frameworks, such as vLLM, TensorRT-LLM, and TGI, have achieved extremely high performance. While they have addressed issues like memory fragmentation, attention computation bottlenecks, and low GPU utilization by introducing optimization techniques such as paging attention mechanisms, flash attention mechanisms, and sequential batch processing, enabling efficient distributed inference on single-type hardware clusters (such as homogeneous NVIDIA GPU clusters or homogeneous AMD GPU clusters), when adapting to heterogeneous hardware, they generally adopt a "multi-backend isolation" architectural paradigm. This means that each hardware ecosystem maintains an independent, deeply optimized, and parallel software backend, allowing the framework to select and activate one backend through a detection mechanism at startup or runtime. However, different backends are isolated from each other and cannot work collaboratively during runtime.

[0004] Although the existing LLM inference frameworks have superior performance in homogeneous hardware environments, their fundamental defects in the "multi-backend isolation" architecture are becoming increasingly prominent in the context of the increasingly diversified and fragmented hardware ecosystem. These defects are mainly reflected in the following aspects: (1) Lack of heterogeneous collaborative computing capabilities. Existing frameworks cannot coordinate and mix computing nodes from different vendors in a single distributed inference task. For example, users cannot run one part of an LLM model on an NVIDIA A100 GPU and another part on a domestic Suiyuan GCU. This results in the inability of the purchased mixed computing assets in enterprises to form a unified resource pool. Valuable heterogeneous computing power is idle because it cannot work collaboratively, resulting in a serious computing power island problem. (2) Due to the extremely high cost of adapting new hardware to mainstream frameworks and the lack of collaborative computing capabilities to revitalize existing heterogeneous assets, when expanding computing power, enterprises tend to continue to purchase hardware from the same supplier in order to ensure ecosystem compatibility and management convenience. This forms a de facto vendor lock-in, which is not conducive to the healthy and sustainable development of the entire industry. (3) High maintenance costs. The "multi-backend isolation" paradigm requires repetitive and costly development, porting and verification work for each new hardware. When a new key optimization occurs, an independent adaptation is required for all backends, which greatly increases the maintenance burden.

[0005] In summary, existing LLM inference frameworks cannot perform collaborative computing in heterogeneous hardware environments across vendors, resulting in wasted computing resources, vendor lock-in, and high maintenance costs. Therefore, there is an urgent need for a large language model inference system that can uniformly manage, schedule, and efficiently utilize heterogeneous computing resources across vendors.

[0006] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solutions of the present invention, and does not imply that the relevant information is necessarily prior art. In the absence of evidence indicating that the relevant information was disclosed before the filing date of this invention, the relevant information should not be considered prior art. Summary of the Invention

[0007] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed heterogeneous large language model reasoning system and reasoning method.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] According to a first aspect of the present invention, a distributed heterogeneous large language model inference system is proposed. The system is used to perform large language model inference tasks. The large language model includes multiple computational layers. The system includes an application interface module, a scheduling module, and multiple heterogeneous computational modules arranged in a pipelined manner. The application interface module is configured with multiple application interfaces, each corresponding to a computational mode. Each computational mode is configured with multiple computational mode implementation strategies. Each computational mode implementation strategy is adapted to a computational module with a specific structure. Each application interface can be called by all computational modules to implement the computational mode adapted to its own structure. The scheduling module is used to perform large language model inference tasks according to the large language model. The language model's structure includes a model scheduling scheme and the periodic packaging of multiple inference tasks into task packages. The model scheduling scheme indicates one or more consecutive computation layers to be allocated to each computation module. Each computation module is configured to: schedule the allocated computation layers according to the model scheduling scheme, and call the computation mode implementation strategy that matches its own structure from the application interface module through the corresponding application interface. All computation modules that have completed computation layer scheduling and computation mode implementation strategy invocation are used to obtain the task packages packaged by the scheduling module, and sequentially perform inference on each inference task in the task package in a pipelined parallel manner to obtain the inference result of each inference task.

[0010] Optionally, the scheduling module is configured to: construct a model scheduling scheme by: obtaining the structural parameters of the large language model, including the number of computation layers, the computational load of each computation layer, and the amount of data to be transmitted between two adjacent computation layers; obtaining the parameters of each computation module, including the computational throughput of each computation module and the communication bandwidth between each computation module and its two adjacent computation modules; constructing a processing time calculation function based on the structural parameters of the large language model and the parameters of each computation module, the processing time calculation function being used to determine the processing time required for a computation module to process all allocated computation layers, the processing time including computation time and data transmission time, wherein the computation time is the time required for the computation module to complete the computation of all allocated computation layers, and the data transmission time includes the time required for the computation module to receive data from the previous computation device and send data to the next device; and using a dynamic programming method based on the constructed processing time calculation function to determine the model scheduling scheme with the minimum pipeline cycle.

[0011] Optionally, the processing time calculation function is:

[0012]

[0013] in, This indicates that the first part of the large language model will be... To the Layer computation layer assigned to the first When the first calculation module is used, the first The processing time required for each computing module and This indicates that the first [level] in the large language model will be [represented in sequence]. arrive Layer computing and layer scheduling up to the 1st When the first calculation module is used, the first The computation time and data transmission time required for each computing module Representing the first in a large language model The computational cost of each layer is calculated. Indicates the first The computational throughput of each computing module The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers Indicates the first The computing module and the first Communication bandwidth between computing modules Indicates the first The computing module and the first Communication bandwidth between computing modules.

[0014] Optionally, the dynamic programming method is:

[0015]

[0016] in, This represents the minimum pipeline cycle that can be achieved by scheduling the first i-th computational layers of a large language model to the first j-th computing devices. This indicates the front of the large language model Layer computing and layer scheduling to the front The minimum pipeline cycle that a computing device can achieve. This represents the function for calculating processing time. This indicates that the first part of the large language model will be... To the Layer computation layer assigned to the first When the first calculation module is used, the first The processing time required for each computing module.

[0017] Optionally, all computing modules are configured such that, in every two adjacent computing modules, the output data of the last computing layer allocated to the preceding computing module is used as the input data of the first computing layer allocated to the following computing module.

[0018] Optionally, the system further includes a server configured to: receive inference requests initiated by users, encapsulate each inference request into an inference task and send each inference task to the scheduling module; and periodically feed back the inference results of the inference tasks to the corresponding users.

[0019] Optionally, the scheduling module is configured with an input queue and an output queue. The scheduling module is configured to: continuously receive inference tasks sent by the server, convert the inference tasks into machine language adapted to the input format of the large language model and store them in the input queue; periodically package one or more inference tasks in the input queue into a task package; obtain the inference results in the calculation module and store the inference results in the output queue.

[0020] According to a second aspect of the present invention, a model reasoning method is proposed, the method comprising: acquiring a target reasoning request and a distributed heterogeneous large language model reasoning system as described in the first aspect of the present invention; and processing the target reasoning request using the acquired distributed heterogeneous large language model reasoning system to obtain a reasoning result of the target reasoning request.

[0021] Compared with the prior art, the advantages of the present invention are as follows:

[0022] This invention achieves collaborative scheduling of heterogeneous hardware by constructing an interface system adapted to heterogeneous devices, thereby breaking down computing power silos, revitalizing enterprises' existing hybrid computing power assets, eliminating dependence on a single vendor for large language model inference, and reducing software adaptation and maintenance costs. In addition, this invention achieves load balancing of heterogeneous clusters based on an adaptive model scheduling strategy, thereby effectively improving the utilization rate of the computing power pool and increasing system throughput, providing a feasible path for unified management and efficient utilization of cross-vendor heterogeneous computing resources. Attached Figure Description

[0023] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0024] Figure 1 is a schematic diagram of a distributed heterogeneous large language model inference system according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of the construction method of the model scheduling scheme according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] As mentioned in the background section, existing LLM inference frameworks cannot perform collaborative computing in heterogeneous hardware environments across vendors, resulting in wasted computing resources, vendor lock-in, and high maintenance costs.

[0028] The inventors conducted source code-level analysis of leading inference frameworks such as vLLM and TensorRT-LLM, and deeply studied their high-performance implementation principles. They discovered that the high performance of these frameworks is inseparable from the deep integration of their software stack and underlying hardware. Their core flaw lies in the tight coupling between the upper-level inference logic (such as model structure and scheduling strategies) and the underlying hardware-specific implementation (such as CUDAKernel and the NCCL communication library). Existing frameworks, based on an "if-else" backend selection mechanism, are essentially a static "multiple-choice" selection rather than a dynamic fusion of "multi-party collaboration." This discovery reveals the technical challenge of this invention: how to break the tight coupling between software and hardware without sacrificing excessive performance, thus decoupling the upper-level inference logic from the underlying heterogeneous hardware.

[0029] To address the aforementioned issues, the inventors propose introducing an intermediate abstraction layer into the existing LLM inference framework to overcome the problem of tight coupling between hardware and software. Specifically, the core computation and state management logic in LLM inference is abstracted at a higher level, constructing a "Hardware Abstraction Layer (HAL)" for the LLM inference domain. This HAL defines a unified, hardware-independent interface specification to describe the computational logic of core operators (such as Attention) and the management methods of key states (such as KVCache). The upper-layer inference framework is developed entirely based on this abstract interface, while hardware vendors can provide pluggable backend implementations that conform to this interface specification. Based on this idea, the inventors further propose a design scheme for a distributed heterogeneous large language model inference system. This scheme achieves load balancing in a heterogeneous environment by automating performance evaluation and modeling of the heterogeneous cluster. The scheduler can perceive the actual computational capabilities of each computation module, thereby performing asymmetric model partitioning during model loading.

[0030] According to an embodiment of the present invention, a distributed heterogeneous large language model inference system is proposed. Referring to Figure 1, the system includes a server, an application interface module (function factory), a scheduling module (engine), and multiple computation modules (workers) arranged in a pipelined manner. The server is responsible for receiving inference requests initiated by users, encapsulating the inference requests into inference tasks, and feeding back the inference results to the corresponding users. The application interface module is a key module for achieving hardware-software decoupling, and it is configured with an interface system adapted to heterogeneous devices. The scheduling module is used to allocate the computation layer of the large language model to each computation module and is responsible for distributing inference tasks and collecting inference results. The computation modules process inference tasks based on the scheduled computation layer, and each computation module is bound to a specific hardware device. To facilitate understanding of the present invention, each module in the distributed heterogeneous large language model inference system will be further described below.

[0031] According to one embodiment of the present invention, the server serves as the interaction interface between the system and the user. It receives inference requests initiated by the user, encapsulates each inference request into an inference task, sends each inference task to the scheduling module, and periodically retrieves inference results from the scheduling module and sends the retrieved inference results to the corresponding user. Specifically, when initiating an inference request, the server receives an HTTP request message containing the inference request from the user, parses the HTTP request message to obtain the inference request, completes the inference request with parameters such as a text prompt, maximum generation length, and temperature coefficient, assigns it a unique request ID, encapsulates the processed inference request into a standardized inference task object, submits the task object to the scheduling module, and, after the calculation module completes the inference task and feeds back the inference results to the scheduling module, the server retrieves the inference results from the scheduling module and feeds back the inference results to the user.

[0032] According to one embodiment of the present invention, the application interface module is configured with multiple application interfaces, each corresponding to a computation mode. Each computation mode is configured with multiple computation mode implementation strategies, and each computation mode implementation strategy is adapted to a computation module with a specific structure. Each application interface can be called by all computation modules to adapt to the computation mode implementation strategy of its own structure. Specifically, the application interface module (function factory) is configured with an interface system adapted to heterogeneous image processors. This system includes multiple function interfaces, each corresponding to multiple function implementations adapted to image processors of different architectures. Specifically, the application interface module is configured with a standardized, hardware-independent interface system. This system includes: operator function interfaces (such as IATtentionFunction), which are used to encapsulate core computations such as Attention and matrix multiplication; state manager interfaces (such as IKVCacheManager), which are used to abstract the lifecycle management of key states such as PagedKVCache; and communication primitive interfaces (ICommunicator). Backend implementations of different hardware (such as CUDA-based FlashAttention and operators based on Suiyuan GCU) are encapsulated into dynamic link libraries (.so) plugins that conform to these interfaces. Furthermore, each interface is configured with multiple function implementations adapted to different image processor architectures. This architecture achieves complete decoupling between upper-layer inference logic and lower-layer hardware implementation, greatly reducing adaptation costs and time, and improving the system's openness and scalability.

[0033] According to one embodiment of the present invention, the scheduling module is used to construct a model scheduling scheme based on the structure of a large language model and to periodically package multiple inference tasks into task packages. The model scheduling scheme indicates one or more consecutive computing layers to which each computing module is assigned. Specifically, the scheduling module pre-obtains the computing layer parameters of the large language model and the parameters of each computing module, and then constructs a model scheduling scheme based on the computing layer parameters of the large language model and the parameters of the computing modules. This scheme asymmetrically allocates the computing layers of the large language model to various computing devices with different aggregate computing power, thereby ensuring that the processing time of each computing device is as balanced as possible.

[0034] According to an embodiment of the present invention, referring to Figure 2, the steps of constructing a model scheduling scheme include: obtaining the structural parameters of a large language model, including the number of computational layers, the computational load of each computational layer, and the amount of data to be transmitted between two adjacent computational layers; obtaining the parameters of each computational module, including the computational throughput of each computational module and the communication bandwidth between each computational module and its two adjacent computational modules; constructing a processing time calculation function based on the structural parameters of the large language model and the parameters of each computational module, wherein the processing time calculation function is used to determine the processing time required for a computational module to process all allocated computational layers, the processing time including computational time and data transmission time, wherein the inference time is the time required for a computational module to complete the inference of the computational layers allocated by the model scheduling scheme, the inference time is determined by the sum of the computational load of all computational layers allocated to the computational module and the computational throughput of the computational module, the data transmission time is the time required for a computational module to receive data from the previous device and send data to the next device, the data transmission time is determined by the amount of data to be transmitted between computational modules and the bandwidth between computational modules; and determining the model scheduling scheme with the minimum pipeline cycle based on the time calculation function using a dynamic programming algorithm.

[0035] According to one embodiment of the present invention, the processing time calculation function is:

[0036]

[0037] in, This indicates that the first part of the large language model will be... To the Layer computation layer assigned to the first When the first calculation module is used, the first The processing time required for each computing module and This indicates that the first [level] in the large language model will be [represented in sequence]. arrive Layer computing and layer scheduling up to the 1st When the first calculation module is used, the first The computation time and data transmission time required for each computing module Representing the first in a large language model The computational cost of each layer is calculated. Indicates the first The computational throughput of each computing module The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers Indicates the communication bandwidth between the th computing module and the th computing module, Indicates the communication bandwidth between the

[0038] It should be understood that the pipeline cycle refers to the global cycle of the entire pipeline, which is determined by the device with the longest processing time (processing cycle) among all computing devices in the entire pipeline. Exemplarily, assuming that the processing times (processing cycles) of three computing devices are 10s, 6s, and 8s respectively, then the pipeline cycle of these three computing devices is 10s.

[0039] According to an embodiment of the present invention, in order to make the processing times of all computing devices as balanced as possible, in the present invention, the optimization goal of the dynamic programming algorithm is to find a large language model scheduling scheme that makes the maximum processing time among the processing times of all computing units the shortest. The dynamic programming algorithm needs to determine the state and the state transition equation. Among them, the present invention defines a two-dimensional DP array, the number of rows of this array is the number of computing layers of the large language model, and the number of columns is the number of computing modules. dp[i][j] is an element in the two-dimensional DP array, which represents the minimum pipeline cycle that can be achieved by dividing the first i layers of the model among the first j computing devices. In order to calculate dp[i][j], it is necessary to find an optimal "cut-off point" p (j - 1 ≤ p < i) for the jth computing unit. This means that the jth computing unit is responsible for processing all computing layers from p + 1 to i, while the first j - 1 computing units are responsible for processing the first p computing layers. The processing time of the jth stage is , that is, the time-consuming for allocating the layers from p + 1 to i to the jth hardware stage. The optimal pipeline cycle of the first j - 1 stages is dp[p][j - 1]. Therefore, when the cut-off point is p, the cycle of the entire pipeline will be the larger value of the two. Traverse all possible cut-off points p to find the cut-off point p that makes this maximum value the smallest. Thus, the state transition equation is:

[0040]

[0041] Among them, represents the minimum pipeline cycle that can be achieved by scheduling the first i layers of the large language model to the first j computing devices, represents the minimum pipeline cycle that can be achieved by scheduling the first layers of the large language model to the first computing devices, represents the processing time calculation function, represents from arrive The computational layer is assigned to the first When the first calculation module is used, the first The processing time required for each computation module is calculated. Based on the state transition equation, the value of each element in the two-dimensional DP array is calculated, ultimately obtaining the globally optimal pipeline cycle that can be achieved by all computation layers and all computation modules of the large language model. At the same time, the optimal decision of each state (i.e., the position of the split point p) is recorded during the computation process. After the algorithm ends, a specific, asymmetric model scheduling scheme can be constructed by backtracking.

[0042] According to one embodiment of the present invention, referring again to Figure 1, in this invention, the scheduling module (engine) is configured with an input queue and an output queue. The scheduling module continuously receives inference tasks sent by the server, converts the inference tasks into machine language adapted to the input format of the large language model and stores them in the input queue, and periodically packages one or more inference tasks in the input queue into a task package, and then saves the task package to shared memory for the computing module to retrieve, thereby achieving low-latency, high-throughput task distribution. Furthermore, after the computing module completes the inference task, it obtains the inference result and stores the inference result in the output queue so that the server can retrieve the inference result.

[0043] According to an embodiment of the present invention, each computing module is configured to: schedule the allocated computing layer according to the model scheduling scheme, and call the computing mode implementation strategy that matches its own structure from the application interface module through the corresponding application interface. In every two adjacent computing modules, the output data of the last computing layer allocated to the previous computing module is used as the input data of the first computing layer allocated to the next computing module. All computing modules that have completed the computing layer scheduling and computing mode implementation strategy call are used to obtain the task package packaged by the scheduling module, and use a pipelined parallel approach to sequentially infer each inference task in the task package to obtain the inference result of each inference task.

[0044] According to an embodiment of the present invention, the present invention proposes a reasoning method based on the above-mentioned distributed large language model reasoning system. The method includes: obtaining a target reasoning request and the above-mentioned distributed heterogeneous large language model reasoning system, and then using the obtained distributed heterogeneous large language model reasoning system to process the target reasoning request to obtain the reasoning result of the target reasoning request.

[0045] To facilitate understanding of how the distributed large language model reasoning system proposed in this invention performs task reasoning, the following will use a single reasoning request as an example to illustrate the process from the task request entering the reasoning system to the return of the final reasoning result. The reasoning process includes a first step, a second step, a third step, and a fourth step, each of which will be explained below.

[0046] The first step is system initialization and adaptive model loading.

[0047] An adaptive model partitioning strategy is constructed, namely, a model scheduling scheme is built, and the computational layers of a large language model are asymmetrically partitioned into a heterogeneous hardware cluster based on the model scheduling scheme. The heterogeneous hardware cluster consists of multiple heterogeneous computing devices. Specifically, this includes the following steps: loading a hardware topology cost matrix, which provides the actual performance data of each computing device in the heterogeneous hardware cluster, including the computational throughput (TFLOPS) of each computing device and the communication bandwidth between each computing device and its neighboring devices; the engine (scheduling module) generates an optimal model deployment scheme based on the hardware topology cost matrix and the structure of the large language model. This scheme defines in detail which layers of the large language model should be assigned to which computing devices (workers); the engine distributes the model scheduling scheme to all computing devices (workers), and each computing device (worker) loads the computational layers assigned to it and the corresponding model weights according to the received instructions, and queries the function factory for the required core computational functions. For example, a worker bound to an NVIDIA GPU will request a function implementation with a device type of CUDA. A computation module that has completed loading the computation layer, each module possessing only a portion of the model, is ready to execute computations on that portion of the model. After this step, the entire large language model is distributed across the heterogeneous cluster in a pipelined parallel manner, and the system enters a standby state.

[0048] The second step is the reception and scheduling of inference requests.

[0049] When a user initiates an inference request (e.g., via an API call), the process is as follows: The server receives the HTTP request from the user. The HTTP request typically includes parameters such as the text prompt to be completed, the maximum generation length, and the temperature coefficient. The server parses the HTTP request, assigns it a unique request ID, and encapsulates it into a standardized inference task object. This task object is then submitted to the engine. The engine places the inference task object into the input queue. The scheduler in the engine receives the inference task object from the server. It tokenizes the task's text prompt, converting it into a sequence of input TokenIDs, and allocates a logical memory block for this request to store the key-value cache (KVCache) generated during subsequent generation. Based on a "continuous batch processing" strategy, the scheduler combines this new request with other requests waiting or being processed in the queue, forming a micro-batch ready to be executed in the next computation step. The output is a micro-batch containing one or more requests ready for execution, along with their corresponding input TokenIDs.

[0050] The third step is the reasoning process.

[0051] The first computing device receives the inference task, calls functions adapted to its own hardware structure to complete the computation of the inference task on its assigned computing layer, and sends the computation result to the next computing device; the intermediate computing device receives the computation result from the previous computing device, calls functions adapted to its own hardware structure to complete the computation on its assigned computing layer; the last computing device receives the computation result from the penultimate computing device, calls functions adapted to its own hardware structure to complete the computation on its assigned computing layer, and feeds back the final computation result to the engine.

[0052] The fourth step is to return the reasoning result.

[0053] The engine compiles the inference results into a human-readable text string. The final text string is then placed in an output queue, awaiting retrieval by the server. The server periodically retrieves the completed inference results from the engine's output queue, encapsulates the final text into an HTTP response, and sends it to the user who initially initiated the request.

[0054] This invention achieves comprehensive performance enhancement and scalability optimization of the LLM inference system through three core technologies. Its beneficial effects are mainly reflected in the following aspects: First, it constructs a standardized, hardware-independent interface system, achieving complete decoupling between upper-layer inference logic and lower-layer hardware, significantly reducing the adaptation cost and cycle of new heterogeneous hardware, and significantly improving the system's openness and scalability. Second, it generates an accurate hardware topology cost matrix through automated evaluation of heterogeneous hardware topology, providing reliable data support for scheduling decisions, allowing the system to accurately perceive the real performance of each node rather than relying on theoretical peak values. Third, it utilizes an adaptive model partitioning and scheduling strategy for heterogeneous pipeline parallelism, asymmetrically distributing the model layer to each pipeline stage, achieving efficient load balancing, minimizing pipeline bubbles, fully releasing the comprehensive performance of the heterogeneous computing pool, and thus maximizing the overall system throughput.

[0055] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0056] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0057] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0058] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A distributed heterogeneous large language model inference system, the system being used to perform large language model inference tasks, the large language model comprising multiple computational layers, characterized in that, The system includes an application interface module, a scheduling module, and multiple heterogeneous computing modules arranged in a pipelined manner. The application interface module is configured with multiple application interfaces, each corresponding to a computing mode. Each computing mode has multiple implementation strategies, and each implementation strategy is adapted to a computing module with a specific structure. Each application interface can be called by all computing modules to adapt to its own structure's implementation strategy. The scheduling module is used to construct a model scheduling scheme based on the structure of the large language model and periodically package multiple inference tasks into task packages. The model scheduling scheme indicates one or more consecutive computing layers allocated to each computing module. Each computing module is configured to: schedule the allocated computing layers according to the model scheduling scheme; and call the computing mode implementation strategy matching its own structure from the application interface module through the corresponding application interface. All computing modules that have completed computing layer scheduling and computing mode implementation strategy invocation are used to obtain the task packages packaged by the scheduling module and sequentially perform inference on each inference task in the task package in a pipelined parallel manner to obtain the inference result of each inference task.

2. The system according to claim 1, characterized in that, The scheduling module is configured to construct a model scheduling scheme in the following manner: obtain the structural parameters of the large language model, including the number of computation layers, the computational amount of each computation layer, and the amount of data to be transmitted between each two adjacent computation layers; obtain the parameters of each computation module, including the computational throughput of each computation module and the communication bandwidth between each computation module and its two adjacent computation modules. A processing time calculation function is constructed based on the structural parameters of the large language model and the parameters of each computing module. The processing time calculation function is used to determine the processing time required for the computing module to process all the assigned computing layers. The processing time includes computing time and data transmission time. The computing time is the time required for the computing module to complete the computing of all the assigned computing layers. The data transmission time includes the time required for the computing module to receive data from the previous computing device and send data to the next device. Based on the constructed processing time calculation function, a dynamic programming method is used to determine the model scheduling scheme with the minimum pipeline cycle.

3. The system according to claim 2, characterized in that, The processing time calculation function is: in, This indicates that the first part of the large language model will be... To the Layer computation layer assigned to the first When the first calculation module is used, the first The processing time required for each computing module and This indicates that the first [level] in the large language model will be [represented in sequence]. arrive Layer computing and layer scheduling up to the 1st When the first calculation module is used, the first The computation time and data transmission time required for each computing module Representing the first in a large language model The computational cost of each layer is calculated. Indicates the first The computational throughput of each computing module The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers The first character representing the large language model Layer and First The amount of data that needs to be transferred between computing layers Indicates the first The computing module and the first Communication bandwidth between computing modules Indicates the first The computing module and the first Communication bandwidth between computing modules.

4. The system according to claim 3, characterized in that, The dynamic programming method is as follows: in, This represents the minimum pipeline cycle that can be achieved by scheduling the first i-th computational layers of a large language model to the first j-th computing devices. This indicates the front of the large language model Layer computing and layer scheduling to the front The minimum pipeline cycle that a computing device can achieve. This represents the function for calculating processing time. This indicates that the first part of the large language model will be... To the Layer computation layer assigned to the first When the first calculation module is used, the first The processing time required for each computing module.

5. The system according to claim 1, characterized in that, All computing modules are configured such that, in every two adjacent computing modules, the output data of the last computing layer assigned to the preceding computing module is used as the input data of the first computing layer assigned to the following computing module.

6. The system according to claim 1, characterized in that, The system also includes a server configured to: receive inference requests initiated by users, encapsulate each inference request into an inference task and send each inference task to the scheduling module; and periodically feed back the inference results of the inference tasks to the corresponding users.

7. The system according to claim 6, characterized in that, The scheduling module is configured with an input queue and an output queue. The scheduling module is configured to continuously receive inference tasks sent by the server, convert the inference tasks into machine language that is adapted to the input format of the large language model, and store them in the input queue. Periodically package one or more inference tasks in the input queue into a task package; retrieve the inference results from the computation module and store the inference results in the output queue.

8. A model reasoning method, characterized in that, The method includes: acquiring a target inference request and a distributed heterogeneous large language model inference system as described in any one of claims 1-7; and using the acquired distributed heterogeneous large language model inference system to process the target inference request to obtain the inference result of the target inference request.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method of claim 8.

10. An electronic device, characterized in that, include: One or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method of any one of claims 8 by executing the executable instructions.

Images