Large model inference acceleration method and system based on dynamic management of model parameters

Abstract

The application discloses a large model inference acceleration method and system based on dynamic management of model parameters, and belongs to the technical field of large models. The method comprises the following steps: based on PCIE running bandwidth and GPU running bandwidth, the number of layers K of each group of model parameters performing inference on the GPU is calculated; after dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory, and the dynamic parameters are saved in the CPU memory, so as to generate an additional key-value cache and a shared physical memory area of the dynamic parameters in the GPU memory; when performing model inference of the i-th group on the GPU, the i+1-th group of dynamic parameters is loaded into the shared physical memory area through PCIE; wherein, the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters. The application can utilize the idle CPU memory and PCIE bandwidth, reduce the total occupation of model parameters in the memory, and reduce the additional overhead caused by dynamic loading of the model through parallelization of the inference calculation and dynamic parameter loading pipeline.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large models, and particularly relates to a large model inference acceleration method and system based on dynamic management of model parameters. BACKGROUND

[0002] Large language models (LLMs) have demonstrated superior performance in natural language processing (NLP) tasks. Models such as GPT4, Llama3, and Deepseek have achieved remarkable results in numerous applications such as machine translation, text generation, and question-answering systems. However, as the model size continues to grow, the inference process in existing LLM inference systems requires more GPU computing resources and GPU memory.

[0003] The GPU memory requirement of large-scale pre-trained models is mainly composed of two parts: model parameters and KV CACHE. Model parameters include hundreds of millions to tens of billions of parameters. For example, the GPT-3 model contains 175 billion parameters, while the BERT-large model contains approximately 340 million parameters. These parameters need to be loaded into the GPU memory during the inference process to ensure that the model can run normally and perform efficient calculations on the GPU. Although a single or multiple GPU cards can load these parameters on existing commercial GPUs (such as NVIDIA V100 with 32GB memory or NVIDIA A100 with 40GB or 80GB memory), they will occupy a large portion of the memory, leaving limited memory resources for KV CACHE.

[0004] KV CACHE is used to store intermediate calculation results generated during model inference. The amount of these cached data increases quadratically with the length of the input token (word unit). This causes the memory requirement to rapidly expand with the increase in input sequence length, especially when generating long context, the number of batches that can be processed in a single inference is limited, and the bottleneck is limited by the available memory of the system, further limiting the number of batches that can be processed. Limited batch data leads to low utilization of GPU computing resources, directly affecting the throughput of inference and reducing inference efficiency.

[0005] In the case of limited memory, the number of batches that can be processed in a single inference is very limited. For example, on a GPU with 32GB of memory, only a small number of samples (such as a few or tens) can be processed simultaneously, and the parallel computing capabilities of the GPU cannot be fully utilized, resulting in low utilization of GPU computing resources. The reduction in the number of batches directly affects the throughput and efficiency of inference, limiting the inference speed and making it difficult to fully utilize the potential of these large models.

[0006] Building upon this foundation, researchers have undertaken various explorations. For instance, Sheng Y et al. proposed FlexGen (Sheng Y, Zheng L, Yuan B, et al. Flexgen: High-throughput generative inference of large language models with a single GPU [C] / / International Conference on Machine Learning. PMLR, 2023:31094-31116.), which offloads some parameters to CPU memory according to the proportion of parameters within a layer. When these parameters need to be computed, they are moved from CPU memory to GPU memory. The main purpose of FlexGen is to run large language models exceeding the capacity of GPU memory, enabling models that would otherwise be impossible to run on machines with limited GPU memory. However, its inference latency far exceeds that of in-memory inference, making it difficult to meet the service level objective (SLO) of LLM inference systems. Summary of the Invention

[0007] This invention proposes a method and system for accelerating large model inference based on dynamic management of model parameters. It can utilize idle CPU memory and PCIe bandwidth to reduce the total occupation of model parameters in video memory. Furthermore, by parallelizing inference computation and dynamic parameter loading pipelines, it reduces the additional overhead caused by dynamic model loading.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following:

[0009] A method for accelerating large model inference based on dynamically managed model parameters, the method comprising:

[0010] Based on the PCIe bandwidth and GPU bandwidth, calculate the number of layers K for each group of inference execution on the GPU for a large model;

[0011] After dividing each set of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into GPU memory and the dynamic parameters are stored in CPU memory to generate an additional key-value cache in GPU memory and a shared physical memory area for the dynamic parameters.

[0012] When performing model inference for the i-th group on the GPU, the dynamic parameters of the (i+1)-th group are loaded into the shared physical memory region via PCIe; wherein, the model inference time for the i-th group is comparable to the time for loading the dynamic parameters of the (i+1)-th group.

[0013] Furthermore, the calculation of the number of layers K for each group of inference execution on the GPU based on PCIe bandwidth and GPU bandwidth includes:

[0014] The PCIe operating bandwidth is obtained based on the data transfer time from the CPU to the GPU. 带宽 ;

[0015] The GPU operating bandwidth is obtained based on the data transfer time from GPU memory to GPU L2 cache. 带宽 ;

[0016] Calculate the number of layers for each group of inference operations on a large model on a GPU.

[0017] Furthermore, based on the number of layers K, each set of model parameters is divided into static parameters and dynamic parameters, including:

[0018] Get the parameter size M of one layer of a large model 单层模型参数 ;

[0019] Calculate the size M of the static parameters. 静态参数 = (Kt)×M 单层模型参数 0 <t<K;

[0020] Calculate the parameter size M of the dynamic parameters 动态参数 =t×M 单层模型参数 ;

[0021] Based on the size M of the static parameters 静态参数 and the size M of the dynamic parameters 动态参数 Each set of model parameters is divided into static parameters and dynamic parameters.

[0022] A large model inference acceleration system based on dynamic management of model parameters, the system comprising:

[0023] The computation module is used to calculate the number of layers K for each group of inference execution on the GPU for a large model, based on the PCIe bandwidth and GPU bandwidth.

[0024] The loading module is used to divide each group of model parameters into static parameters and dynamic parameters according to the number of layers K, load the static parameters into GPU memory, and save the dynamic parameters in CPU memory, so as to generate an additional key-value cache in GPU memory and a shared physical memory area for dynamic parameters.

[0025] The inference module is used to load the (i+1)th group of dynamic parameters into the shared physical memory area via PCIe when performing model inference for the i-th group on the GPU; wherein the model inference time for the i-th group is comparable to the time for loading the (i+1)th group of dynamic parameters.

[0026] An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the large model inference acceleration method based on dynamic management model parameters as described above.

[0027] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the large model inference acceleration method based on dynamically managed model parameters as described above.

[0028] The technical solutions provided in this disclosure have at least the following beneficial effects:

[0029] 1) This invention employs a multi-layered parameter dynamic loading technique. Through a specific arrangement method, this invention loads multi-layered model parameters into the same physical location, utilizing idle CPU memory and PCIe bandwidth. This method effectively reduces the GPU memory usage of model parameters, especially when processing large-scale models. The saved GPU memory is used to store the KV cache, thereby increasing the number of batches processed by the system and improving the throughput of the inference system.

[0030] 2) This invention automatically generates a dynamic memory loading scheme. This invention analyzes the current PCIe bandwidth and GPU memory bandwidth of the hardware, generates a dynamic memory loading strategy based on the actual available bandwidth, adjusts the additional overhead caused by memory unloading, and optimizes memory usage efficiency. By automatically analyzing and generating dynamic memory loading strategies, efficient management and optimization of memory resources can be achieved, significantly improving memory utilization, reducing the additional overhead caused by memory unloading, and enhancing the overall performance of the system when performing large model inference.

[0031] 3) This invention pipelines inference computation and dynamic loading. During the inference computation process, this invention pipelines the dynamic parameter loading and computation process, reducing the additional overhead caused by dynamic loading. By parallelizing the dynamic loading and computation process, the model inference process can be made more efficient, reducing waiting time, improving system response speed, and enhancing overall inference efficiency, which helps to achieve faster computation and response in practical applications. Attached Figure Description

[0032] Figure 1 This diagram shows the interconnect structure between CPU memory and GPU video memory via PCIe.

[0033] Figure 2 Flowchart of a method to accelerate large-scale inference using dynamic memory management.

[0034] Figure 3 The transmission scheme is determined based on the PCIe bandwidth and GPU memory bandwidth.

[0035] Figure 4 Static and dynamic parameter management.

[0036] Figure 5 Overlapping flowchart of inference computation and dynamic parameter loading.

[0037] Figure 6 A comparison chart of the throughput of the present invention and the prior art. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to specific embodiments. The embodiments given are only for illustrating the present invention and are not intended to limit the scope of the present invention.

[0039] The large model inference acceleration system based on dynamically managed model parameters of the present invention, such as Figure 1 As shown, the CPU's high-capacity memory is responsible for storing the model's static parameters and dynamically loaded parameters, while the GPU's high-bandwidth video memory is responsible for performing inference calculations and dynamically loading and unloading model parameters in video memory. PCIe bandwidth enables fast data transfer between CPU memory and GPU video memory. Specifically, this invention mainly utilizes idle CPU memory and PCIe bandwidth to dynamically load large model parameters to the GPU at runtime. Through specific orchestration, multi-layered model parameters are loaded into the same physical location, reducing the total memory usage of model parameters. Furthermore, by parallelizing the inference calculation and dynamic parameter loading pipelines, the additional overhead caused by dynamically loading models is reduced.

[0040] The present invention provides a method for accelerating large model inference through dynamic management of model parameters, such as... Figure 2 As shown, it includes the following steps.

[0041] Step 1: Based on the PCIe bandwidth and GPU bandwidth, calculate the number of layers K for each group of inference operations on the GPU for a large model.

[0042] In one embodiment, the present invention first measures the PCIe operating bandwidth by measuring the transfer time of 4MB of data from the CPU to the GPU. 带宽 Furthermore, by measuring the transfer time of 4MB of data from GPU memory to GPU L2 cache, the GPU bandwidth was determined. 带宽 Next, obtain the parameter size M of one layer of the model to be loaded. 单层模型参数 .

[0043] Subsequently, in order to overlap the model parameter loading time from CPU to GPU and the inference execution time of the K-layer model on the GPU (the total number of models is N layers), Group reasoning, performing K levels of reasoning at a time, such as Figure 3The present invention adjusts the group size to make the data transmission time of each group equal, thus avoiding idle waiting. According to formula (1), the group size K is evaluated based on the actual measured model parameter size, PCIe bandwidth, and GPU bandwidth (because large model inference is mainly limited by memory bandwidth, the actual inference time can be approximated by GPU memory).

[0044]

[0045] Step 2: After dividing each set of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory, and the dynamic parameters are stored in the CPU memory, so as to generate a key-value cache in the GPU memory and a shared physical memory area for the dynamic parameters.

[0046] In the model parameter initialization phase, this embodiment follows... The system loads static model parameters into video memory in groups and stores all dynamically loaded parameters in CPU memory. It then reclaims the video memory used for dynamic model parameters, allocating the saved video memory to the KV cache, such as... Figure 4 As shown, the static model parameters of layers 1, 2 and layers 4, 5 have been loaded into GPU memory, while the model parameters of layers 3 and 6 are stored in CPU memory, saving GPU memory. The saved GPU memory is used for KV CACHE, thereby increasing the number of batches during model inference.

[0047] Step 3: When performing model inference for group i on the GPU, load the dynamic parameters of group i+1 into the shared physical memory area via PCIe; the model inference time for group i is comparable to the time for loading the dynamic parameters of group i+1.

[0048] This invention performs grouped model inference based on the generation scheme. During inference, dynamic parameters are loaded from CPU memory to shared physical memory in parallel. After the parameters required for the current inference group are loaded into the memory, inference calculations are performed for each group. The GPU inference of the preceding layers and the data transfer from CPU memory to GPU memory for subsequent layers are synchronized to ensure that the layer parameters required for the next calculation are loaded into the memory before the next GPU inference.

[0049] Specifically, the dynamic parameter loading method described in this invention involves loading the dynamic model parameters required for the next set of inferences from CPU memory to shared physical video memory simultaneously with GPU inference, utilizing the PCIe bandwidth. Since the dynamically loaded parameters are loaded into the same GPU memory, the computation time of the K-layer model in the i-th set is comparable to the time required to load the 1-layer model in the (i+1)-th set during inference. Therefore, the overhead of dynamically loading model parameters overlaps with the computational overhead (e.g., ...). Figure 5As shown, multi-layer model parameters are loaded into the same physical location, reducing the use of shared physical memory.

[0050] Finally, determine whether the reasoning is complete. If not, proceed to the next set of reasoning; otherwise, return the reasoning result.

[0051] The following is a specific experiment to illustrate the method for accelerating large model inference based on dynamically managed model parameters provided by this invention.

[0052] Researchers tested this invention on an RTX 4050, running the opt1.3b model under limited video memory conditions. The baseline lacked a method for dynamically managing video memory. Figure 6 As shown, 0.23GB of dynamic video memory was saved and used for KVCACHE, resulting in a 1.41x increase in throughput with a total memory footprint of 3GB.

[0053] In summary, the multi-layer parameter dynamic loading technology of this invention loads multi-layer model parameters to the same physical location on the GPU memory through a specific arrangement method, such as... Figure 4 As shown, layers 3 and 6 are dynamically loaded from CPU memory to time-sharing shared GPU memory. This utilizes idle and large-capacity CPU memory and free PCIe bandwidth. When layers 3 and 6 sharing GPU memory are requested for inference, parameters are dynamically transferred from CPU memory to designated GPU memory via PCIe, improving the utilization of CPU memory and PCIe bandwidth and expanding the size of GPU memory, which has become a bottleneck in large model inference.

[0054] The automatic dynamic memory loading scheme of this invention analyzes the current hardware's PCIe bandwidth and GPU memory bandwidth (Bandwidth) to obtain, for example... Figure 3 GPU_Memory Bandwidth (GPU memory bandwidth) and PCIe Bandwidth The goal is to adjust these two values: (PCIe memory bandwidth) Figure 3 The size K of the reasoning groups in Time makes Time PCIE (PCIe transfer time) equals Time GPU_Memory (GPU memory transfer time) is reduced, preventing individual data loading from waiting idly. This generates a dynamic memory loading strategy, reducing the overhead of dynamic parameter loading and optimizing memory usage efficiency.

[0055] The pipelined inference calculation and dynamic loading of model parameters of the present invention pipelines the dynamic parameter loading and calculation process during the inference calculation process, thereby reducing the additional overhead caused by dynamic loading.

[0056] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A large model inference acceleration method based on dynamic management of model parameters, characterized in that, The method comprises: Based on the PCIE running bandwidth and the GPU running bandwidth, the number of layers K of each group of model parameters is calculated. After dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory, and the dynamic parameters are saved in the CPU memory, so as to generate an additional key-value cache and a shared physical memory area of the dynamic parameters in the GPU memory; When the model inference of the i-th group is performed on the GPU, the i+1-th group of dynamic parameters is loaded into the shared physical memory area through the PCIE; wherein the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters.

2. The method of claim 1, wherein, The method comprises: According to the data transmission time from the CPU to the GPU, the PCIE running bandwidth PCIE 带宽 ; According to the data transfer time from the GPU memory to the GPU L2 Cache, the GPU running bandwidth GPU 带宽 ; number of layers per group of inference performed by a large language model on a GPU 3. The method of claim 1, wherein, The method comprises: Obtaining parameter size M of one layer of large model 单层模型参数 ; Parameter size M for calculating static parameters 静态参数 = (K - t) x M 单层模型参数 , 0 < t < K; Parameter size M for calculating dynamic parameters 动态参数 = t x M 单层模型参数 ; According to the parameter size M of the static parameter 静态参数 and the parameter size M of the dynamic parameter 动态参数 Each group of model parameters is divided into static parameters and dynamic parameters.

4. A large model inference acceleration system based on dynamically managing model parameters, characterized in that, The system comprises: A calculation module is configured to calculate the number of layers K of each group of model parameters based on the PCIE running bandwidth and the GPU running bandwidth. A loading module is configured to divide each group of model parameters into static parameters and dynamic parameters according to the number of layers K, and then load the static parameters into the GPU memory and save the dynamic parameters in the CPU memory, so as to generate an additional key-value cache and a shared physical memory area of the dynamic parameters in the GPU memory. A reasoning module is configured to load the i+1-th group of dynamic parameters into the shared physical memory area through the PCIE when the model inference of the i-th group is performed on the GPU; wherein the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters.

5. An electronic device, comprising: The electronic device comprises a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the large model inference acceleration method based on dynamic management of model parameters according to any one of claims 1-3.

6. A computer readable storage medium characterized by, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the large model inference acceleration method based on dynamic management of model parameters according to any one of claims 1-5.

Images