A method and system for deploying a large language model on a graphics card with limited computing power

Abstract

The application provides a method and system for deploying a large language model on a graphics card with limited computing power, relating to the technical field of software engineering, and aiming at the characteristics of the target graphics card hardware architecture lacking a dedicated matrix acceleration unit, reconstructing key operators required by the large language model, including: adapting the core computing process of the key operator to be executed by the general computing unit of the graphics card; optimizing the key operator to make it suitable for a computing scenario with an input dimension of 1xN; quantizing the model weights in the key operator by 4-bit integers and using 16-bit floating-point numbers for activation in the forward calculation process; constructing a lightweight inference engine; loading the target large language model and executing the inference task on the target graphics card by using the lightweight inference engine. The application realizes system-level improvement in three dimensions of hardware compatibility, resource utilization, and inference efficiency, and provides a feasible technical path for deploying advanced large models on a wide range of existing and low-power graphics cards.

Description

Technical Field

[0001] This invention relates to the field of software engineering technology, and in particular to a method and system for deploying large language models on graphics cards with limited computing power. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have become a core infrastructure in the field of natural language processing. Currently, mainstream large language model deployment solutions in the industry are highly dependent on specific hardware architectures and software ecosystems. Typically, these models are deployed on a series of high-end graphics processing units (GPUs), making full use of their CUDA parallel computing platform and dedicated hardware acceleration units (such as Tensor Cores) for efficient computation. At the software level, developers generally use open-source high-performance inference engines such as vLLM and SGLang to perform model inference tasks. These engines are usually deeply optimized for the hardware characteristics of GPUs.

[0003] However, when attempting to deploy large language models on some domestically produced graphics cards, especially early low-end models, existing solutions face certain technical limitations. The common approach currently is to directly utilize the open-source inference engine designed for high-performance GPUs, building upon the dedicated development toolchains provided by graphics card manufacturers for specific hardware. This approach reveals the following fundamental flaws in practical applications: First, at the hardware design level, many early domestically produced graphics cards were designed when large language model technology was not yet mature and widespread. Their hardware architecture focused more on traditional graphics rendering or general computing tasks, and generally lacked dedicated hardware acceleration units for key operations in large models, such as large-scale matrix multiplication and attention mechanisms. This resulted in a significant reduction in the execution efficiency of directly ported operators, making it impossible to meet the real-time requirements of actual inference.

[0004] Secondly, in terms of video memory resources, these early graphics cards typically had limited video memory capacity, making it difficult to support the current large-scale advanced language models with a huge number of parameters. They were prone to task failure or a sharp drop in performance during inference due to insufficient video memory.

[0005] Therefore, when existing technical solutions are applied to domestically produced low-end graphics cards with limited computing power and different architectural characteristics, they generally suffer from poor compatibility, low computing efficiency, and insufficient utilization of video memory, which seriously restricts the feasibility and practicality of large language models on such hardware platforms. Summary of the Invention

[0006] This invention overcomes the shortcomings of the prior art and provides a method and system for deploying large language models on graphics cards with limited computing power. This results in system-level improvements in hardware compatibility, resource utilization, and inference efficiency, providing a feasible technical path for deploying advanced large models on a wide range of existing and low-computing-power graphics cards.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: A method for deploying large language models on computing-limited GPUs includes the following steps: Step S1: Addressing the lack of dedicated matrix acceleration units in the target graphics card hardware architecture, reconstruct the key operators required for the large language model; the reconstruction includes: The core computation process of the key operators is adapted to be executed using the general computing units of the graphics card; The key operator is optimized to make it suitable for computational scenarios with an input dimension of 1×N; The model weights in the key operators are quantized using 4-bit integers and activated using 16-bit floating-point numbers during the forward computation process. Step S2: Construct a lightweight inference engine, which integrates the reconstructed key operators and employs at least one of the following optimization mechanisms: Pipeline parallelism is used to distribute different layers or different computational stages of a model to multiple computational units for sequential execution. Tensor parallelism is a mechanism used to split the model's weights or activation tensors into multiple computational units for parallel computation. The pre-filling and decoding are separated architectures, in which the computing resources allocated to the pre-filling stage and the decoding stage are in a ratio of 2:8; Data transmission between pre-filled nodes and decoding nodes is performed using a CPU and GPU collaboration approach. Page attention mechanism, used to efficiently manage key-value caching in attention calculation; The prefix caching mechanism is used to cache and reuse the computation results of repeated prefix parts in the input sequence; Step S3: Using the lightweight inference engine, load the target large language model and execute the inference task on the target graphics card.

[0008] Furthermore, the key operator includes at least one of the FlashAttention operator or the FusedMoE operator.

[0009] Furthermore, in step S1, the computation task of the key operator is transferred from the missing Tensor Core hardware unit to be executed through the CUDA Core or OpenCL computing unit.

[0010] Furthermore, in step S2, the pipeline parallel mechanism is used to solve the problem that the model parameters cannot be loaded into the graphics card memory at once due to their large size.

[0011] Furthermore, in step S2, the CPU and GPU collaborative transmission method is used to temporarily store some intermediate data generated during the decoding stage in system memory to alleviate the immediate pressure on GPU memory.

[0012] Furthermore, in step S2, the page attention mechanism improves video memory utilization efficiency by organizing attention key-value pairs in video memory in a paginated manner.

[0013] Furthermore, in step S2, the prefix caching mechanism reduces redundant computation by identifying and reusing intermediate calculation results corresponding to the same input prefix in different inference requests.

[0014] Furthermore, the combination of pipelined parallelism, tensor parallelism, and CPU-GPU collaborative transmission methods described in step S2 enables the target graphics card to support inference tasks of large language models with a parameter count exceeding its physical memory capacity through dynamic data exchange between system memory and video memory.

[0015] A system for deploying large language models on a computing-limited graphics card, employing any of the methods described above, including: The key operator module provides a key operator implementation suitable for the target graphics card. The key operator, after reconstruction, supports an input dimension of 1×N and uses 4-bit quantization and 16-bit activation. The inference engine module is built on the key operator module and supports at least one of pipeline parallelism, tensor parallelism, pre-filling-decoding separation architecture, CPU-GPU collaborative transmission, page attention mechanism and prefix caching mechanism. The deployment module is used to deploy the inference engine module on the target graphics card to perform inference tasks for large language models.

[0016] Furthermore, the key operator module configures the computational task to be executed by the general-purpose computing unit of the graphics card; The ratio of computation nodes in the pre-filling stage to the decoding stage in the inference engine module is 2:8; The inference engine module achieves data transmission between nodes through CPU-GPU collaborative transmission.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention reconstructs key operators such as FlashAttention, transferring their computational core from missing dedicated hardware units to general-purpose computing units, and performs in-depth optimization for single-batch long sequence inputs, enabling advanced models to run directly on early or domestically produced graphics cards that lack tensor cores, thus breaking through hardware ecosystem dependence.

[0018] A hybrid precision scheme employing 4-bit weighted quantization and 16-bit activation computation significantly reduces GPU memory usage while maintaining inference accuracy. The lightweight inference engine comprehensively utilizes pipelined parallelism, tensor parallelism, and a pre-filling-decoding separation architecture, combined with a CPU-GPU collaborative data transfer mechanism, effectively solving the problem of large model parameters not being able to be loaded into GPU memory at once, achieving dynamic balance and efficient utilization of computational and memory resources.

[0019] By further introducing page attention and prefix caching mechanisms, optimizations are made in two aspects: memory organization and computation reuse, which improves memory utilization efficiency and reduces redundant computation overhead.

[0020] Overall, this invention achieves system-level improvements in three dimensions: hardware compatibility, resource utilization, and inference efficiency, providing a feasible technical path for deploying advanced large models on a wide range of existing and low-performance graphics cards. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and are used together with the embodiments of the invention to explain the invention. They do not constitute a limitation of the invention. In the drawings: Figure 1 This is a system architecture and data processing flowchart of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] like Figure 1 As shown, this invention claims protection for a method for deploying a large language model on a computing-limited graphics card, comprising the following steps: Step S1: Reconstruct key operators based on hardware characteristics Since the target GPU cannot efficiently perform large-scale matrix multiplication operations, which are core to the attention mechanism and feedforward network layers in large language models, the key operators necessary for model inference are first reconstructed. These include: Operator selection and reconstruction strategy: The FlashAttention operator and the FusedMoE operator, which have the greatest impact on performance, are selected as reconstruction objects. The original computation flow, which was designed to utilize Tensor Cores and was based on large-size matrix multiplication (GEMM), is transformed into a loop computation task that is efficiently executed by a series of general-purpose computing units (CUDA Cores) and mainly consists of small-scale matrix-vector multiplication (GEMV) or vector dot product operations.

[0024] Input scenario optimization: Considering that in real-world online inference scenarios, the batch size of a single request is typically 1, and the input is a sequence of length N, this invention performs deep optimization on the reconstructed operators to achieve optimal performance when the input dimension is 1×N. For example, in the reconstruction of FlashAttention, the key-value cache access mode and data loading strategy for single query sequences are optimized.

[0025] Mixed-precision quantization implementation: To reduce memory usage, 4-bit integer quantization is applied to the model weights in the reconstruction operator. Specifically, a grouped quantization method is used to reduce quantization errors. During the activation calculation in forward inference, 16-bit floating-point precision is maintained to ensure computational accuracy and stability. Quantization and dequantization operations are integrated into the operator's computational kernel function to reduce overhead.

[0026] Through the above reconstruction, the FlashAttention and FusedMoE operators, which were originally unable to be compiled and run on this graphics card, were successfully compiled and executed stably through the manufacturer's standard CUDA toolchain.

[0027] Step S2: Build a lightweight inference engine Based on the reconstructed key operators, a lightweight inference engine designed specifically for the target hardware is constructed. This engine integrates multiple resource optimization mechanisms to collaboratively address the dual constraints of computing power and GPU memory.

[0028] Pipeline Parallelism: The 7 billion parameter model is divided into four stages based on network layers. Since the video memory required by a single stage is less than the total video memory of the graphics card, different stages can be loaded into video memory and executed sequentially. During computation, the four stages form a pipeline in time: when the first inference request enters the second stage for computation, the second inference request can enter the first stage for computation, and so on. This mechanism solves the fundamental problem that model parameters cannot all be loaded into video memory at once.

[0029] Tensor parallelism: Within the GPU, the weight matrix of each linear layer of the model is column-wise partitioned and distributed to multiple CUDA Core groups on streaming multiprocessors for parallel computation. The computation results are synchronously reduced within each layer. This fully utilizes the many-core parallelism capabilities of the GPU, accelerating a single forward computation.

[0030] Pre-filling-decoding separation architecture: The inference process is divided into two stages: pre-filling and decoding. The pre-filling stage processes complete input prompts, which is computationally intensive but highly parallelizable; the decoding stage generates output tokens one by one, which is less computationally intensive but has a sequential dependency. This engine allocates approximately 20% of its computational resources to pre-filling nodes and 80% to decoding nodes, a ratio of 2:8. This architecture ensures that computational resources are matched to the computational characteristics of different stages.

[0031] CPU-GPU Collaborative Transfer: In pipelined parallelism and pre-filling-decoding separation architectures, intermediate activation tensors need to be transferred between different computing nodes. This implementation employs CPU-GPU collaborative transfer: after a node completes computation, the data to be passed to downstream nodes is asynchronously copied to a buffer pool in the CPU's main memory using cudaMemcpyAsync; when the downstream node needs it, it is copied from the buffer pool to GPU memory. This approach uses GPU memory as a high-speed cache, avoiding the pressure caused by all intermediate data residing in GPU memory, and especially alleviating the GPU memory growth problem caused by generating long texts during the decoding stage.

[0032] Page Attention Mechanism: To manage the ever-growing key-value cache during decoding, a page attention mechanism is introduced. The video memory is virtually divided into fixed-size pages, for example, each page stores key-value pairs of 256 terms. When a new term is generated, the system allocates new pages to store its key-value pairs on demand, rather than reallocating them in contiguous memory. This approach eliminates memory fragmentation, allowing the key-value cache to be distributed across non-contiguous spaces in the video memory, improving video memory utilization efficiency and supporting the generation of text much longer than the limit of contiguous available video memory space.

[0033] Prefix caching mechanism: A shared prefix cache is set up. When a new inference request arrives, the system first matches its input prompt with the existing prompt prefixes in the cache. If a match is found, the pre-calculated intermediate layer output corresponding to that prefix is ​​directly reused, such as the hidden state after several Transformer layers, skipping redundant calculations for these repeated prefixes. This is particularly suitable for scenarios with multi-turn dialogues and repeated prompt templates, significantly reducing computational overhead.

[0034] Step S3: Model Deployment and Inference Execution After completing the construction of the lightweight inference engine, the target large language model with 7 billion parameters is loaded into the engine using the quantized weights from step S1. Based on the aforementioned optimization mechanism, the engine automatically schedules computational tasks and data flow. Users submit inference requests through the engine's provided API interface, and the engine sequentially performs pre-filling and decoding, ultimately returning the generated text result. On the target GPU in this implementation, this solution successfully achieved stable deployment of the 7 billion parameter model and reached a usable inference speed.

[0035] This invention provides a system for deploying large language models on graphics cards with limited computing power. The system can be implemented in software, hardware, or a combination of both, for example, in an AI inference server or edge computing device.

[0036] The system mainly includes: Key Operator Module: This module stores and manages the implementation code of key operators such as FlashAttention and FusedMoE, refactored for the target GPU. These operators support depth optimization up to 1×N input dimensions and integrate 4-bit integer weight quantization and 16-bit floating-point activation calculation. This module is responsible for executing these operators at runtime by calling the general computing unit through the GPU driver.

[0037] Inference Engine Module: This is the core control and scheduling module of the system. It integrates key operator modules and contains multiple sub-units: Parallel scheduling unit: responsible for implementing pipeline parallelism and tensor parallelism strategies, decomposing the computation graph and scheduling it onto hardware resources.

[0038] Architecture Management Unit: Responsible for maintaining the pre-filled-decoder separation architecture, allocating and managing computing resources between the two in a 2:8 ratio.

[0039] Cooperative Transfer Unit: Manages the buffer pool in the CPU's main memory and controls asynchronous data transfer between the CPU and GPU to support inter-node communication.

[0040] Memory optimization unit: includes a page attention management subunit and a prefix cache management subunit, which are responsible for the paging management of KVCache and the reuse of prefix calculation results, respectively.

[0041] Deployment and Execution Module: This module provides functions such as model loading, weight initialization, and API interfaces. It receives user requests, calls the inference engine module to perform calculations, and returns the results. Simultaneously, it is responsible for interacting with the underlying operating system and graphics card driver to complete the final allocation and release of resources.

[0042] When the system starts, the deployment module first loads the target large language model and key operator modules; the inference engine module then initializes each optimization unit according to the current hardware resources; finally, the system enters the service state, waiting for and processing inference requests.

[0043] In summary, this invention achieves efficient deployment of large language models on resource-constrained hardware through targeted optimization of software and hardware collaboration. Those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these implementation methods without departing from the principles and spirit of this invention.

[0044] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for deploying large language models on graphics cards with limited computing power, characterized in that, Includes the following steps: Step S1: Addressing the lack of dedicated matrix acceleration units in the target graphics card hardware architecture, reconstruct the key operators required for the large language model; the reconstruction includes: The core computation process of the key operators is adapted to be executed using the general computing units of the graphics card; The key operator is optimized to make it suitable for computational scenarios with an input dimension of 1×N; The model weights in the key operators are quantized using 4-bit integers and activated using 16-bit floating-point numbers during the forward computation process. Step S2: Construct a lightweight inference engine, which integrates the reconstructed key operators and employs at least one of the following optimization mechanisms: Pipeline parallelism is used to distribute different layers or different computational stages of a model to multiple computational units for sequential execution. Tensor parallelism is a mechanism used to split the model's weights or activation tensors into multiple computational units for parallel computation. The pre-filling and decoding are separated architectures, in which the computing resources allocated to the pre-filling stage and the decoding stage are in a ratio of 2:8; Data transmission between pre-filled nodes and decoding nodes is performed using a CPU and GPU collaboration approach. Page attention mechanism, used to efficiently manage key-value caching in attention calculation; The prefix caching mechanism is used to cache and reuse the computation results of repeated prefix parts in the input sequence; Step S3: Using the lightweight inference engine, load the target large language model and execute the inference task on the target graphics card.

2. The method according to claim 1, characterized in that, The key operator includes at least one of the FlashAttention operator or the FusedMoE operator.

3. The method according to claim 1, characterized in that, In step S1, the computation task of the key operator is transferred from the missing Tensor Core hardware unit to be executed through the CUDA Core or OpenCL computing unit.

4. The method according to claim 1, characterized in that, In step S2, the pipeline parallel mechanism is used to solve the problem that the model parameters cannot be loaded into the graphics card memory at once due to their large size.

5. The method according to claim 1, characterized in that, In step S2, the CPU and GPU collaborative transmission method is used to temporarily store some intermediate data generated during the decoding stage in system memory to alleviate the immediate pressure on GPU memory.

6. The method according to claim 1, characterized in that, In step S2, the page attention mechanism improves video memory utilization efficiency by organizing attention key-value pairs in video memory in a paginated manner.

7. The method according to claim 1, characterized in that, In step S2, the prefix caching mechanism reduces redundant calculations by identifying and reusing intermediate calculation results corresponding to the same input prefix in different inference requests.

8. The method according to claim 1, characterized in that, The combination of pipelined parallelism, tensor parallelism, and CPU-GPU collaborative transmission in step S2 enables the target graphics card to support inference tasks of large language models with a number of parameters exceeding its physical video memory capacity through dynamic data exchange between system memory and video memory.

9. A system for deploying large language models on a graphics card with limited computing power, employing the method of any one of claims 1 to 8, characterized in that, include: The key operator module provides a key operator implementation suitable for the target graphics card. The key operator, after reconstruction, supports an input dimension of 1×N and uses 4-bit quantization and 16-bit activation. The inference engine module is built on the key operator module and supports at least one of pipeline parallelism, tensor parallelism, pre-filling-decoding separation architecture, CPU-GPU collaborative transmission, page attention mechanism and prefix caching mechanism. The deployment module is used to deploy the inference engine module on the target graphics card to perform inference tasks for large language models.

10. The system according to claim 9, characterized in that, The key operator module configures the computation task to be executed by the general computing unit of the graphics card; The ratio of computation nodes in the pre-filling stage to the decoding stage in the inference engine module is 2:8; The inference engine module achieves data transmission between nodes through CPU-GPU collaborative transmission.

Images