Large model reasoning acceleration method and system based on dynamic management model parameters

Abstract

The invention discloses a large model reasoning acceleration method and system based on dynamic management model parameters, and belongs to the technical field of large models. The method comprises the following steps: based on PCIE operation bandwidth and GPU operation bandwidth, calculating a layer number K of each group of reasoning execution of a large model on a GPU; dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, loading the static parameters to a GPU video memory, and storing the dynamic parameters in a CPU memory so as to generate a shared physical video memory area of an extra key value cache and the dynamic parameters in the GPU video memory; when the model reasoning of the ith group is executed on the GPU, loading the (i + 1) th group of dynamic parameters into the shared physical video memory area through PCIE (Peripheral Component Interface Express); wherein the model reasoning time of the ith group is equivalent to the time for loading the (i + 1) th group of dynamic parameters. According to the method, the idle CPU memory and PCIE bandwidth can be utilized, the total occupation of the model parameters in the video memory is reduced, and the additional overhead caused by dynamically loading the model is reduced through the parallel reasoning calculation and dynamic parameter loading assembly line.

Description

Technical Field

[0001] The present invention belongs to the technical field of large models, and in particular, relates to a large model reasoning acceleration method and system based on dynamic management of model parameters. Background Art

[0002] Language models (LLMs) have demonstrated excellent performance in natural language processing (NLP) tasks. Models such as GPT4, Llama3, and Deepseek have achieved remarkable results in many applications such as machine translation, text generation, and question-answering systems. However, as the scale of the model continues to expand, the demand for GPU computing resources and GPU memory in the reasoning process in the existing LLM reasoning system has also increased.

[0003] The video memory requirements of large-scale pre-trained models mainly consist of two parts: model parameters and KV CACHE (key-value cache). Model parameters include hundreds of millions to tens of billions of parameters. For example, the GPT-3 model contains 175 billion parameters, and the BERT-large model also contains about 340 million parameters. These parameters need to be loaded into the video memory during the inference process to ensure that the model can run normally and perform efficient calculations on the GPU. Although these parameters can be loaded through a single or multiple GPU cards on existing commercial GPUs (such as NVIDIA V100, 32GB video memory, or NVIDIA A100, 40GB or 80GB video memory), they will occupy a large part of the video memory, making the video memory resources reserved for KV CACHE tight.

[0004] KV CACHE (key-value cache) is used to store intermediate calculation results generated during the model inference process. The amount of cached data increases quadratically with the increase in the length of the input token (word unit). This causes the demand for video memory to expand rapidly as the length of the input sequence increases, especially when long contexts are generated. The number of batches that can be processed in a single inference is limited. The bottleneck at this time is limited by the available video memory in the system, which further limits the number of batches that can be processed. The limited batch data leads to low utilization of GPU computing resources, which directly affects the throughput of inference and reduces the efficiency of inference.

[0005] When video memory is limited, the number of batches that can be processed in a single inference is very limited. For example, on a GPU with 32GB of video memory, only a small number of samples (such as a few or a dozen) may be processed at the same time, and the parallel computing capabilities of the GPU cannot be fully utilized, resulting in low utilization of GPU computing resources. The reduction in the number of batches directly affects the throughput and efficiency of inference, limits the speed of inference, and makes it difficult to fully utilize the potential of these large models.

[0006] On this basis, relevant researchers have carried out various explorations. For example, Sheng Y et al. proposed FlexGen (Sheng Y, Zheng L, Yuan B, et al. Flexgen: High-throughput generative inference of large language models with a single gpu [C] / / International Conference on Machine Learning. PMLR, 2023: 31094-31116.), which unloads some parameters to the CPU memory according to the ratio of parameters within the layer. When these parameters need to be calculated, the parameters are moved from the CPU memory to the GPU memory. The main purpose of FlexGen is to be able to run large language models that exceed the memory capacity, and to run models that were originally unable to run on machines with limited memory, but its inference delay far exceeds the inference in the memory, and it is difficult to meet the inference delay SLO (service level objective) of the LLM inference system. Summary of the invention

[0007] The present invention proposes a large model reasoning acceleration method and system based on dynamic management of model parameters, which can utilize idle CPU memory and PCIE bandwidth to reduce the total occupancy of model parameters in video memory, and reduce the additional overhead caused by dynamic loading of models by parallelizing reasoning calculation and dynamic parameter loading pipelines.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0009] A large model reasoning acceleration method based on dynamic management of model parameters, the method comprising:

[0010] Based on the PCIE operating bandwidth and GPU operating bandwidth, calculate the number of layers K of each group of inference execution on the GPU for the large model;

[0011] After dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory, and the dynamic parameters are saved in the CPU memory, so as to generate a shared physical memory area of ​​an additional key-value cache and the dynamic parameters in the GPU memory;

[0012] When executing model inference of the i-th group on the GPU, the i+1-th group of dynamic parameters is loaded into the shared physical video memory area through PCIE; wherein, the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters.

[0013] Furthermore, the method of calculating the number of layers K of each group of reasoning execution of the large model on the GPU based on the PCIE operation bandwidth and the GPU operation bandwidth includes:

[0014] According to the data transmission time from CPU to GPU, the PCIE operating bandwidth PCIE is obtained 带宽 ;

[0015] According to the data transfer time from GPU memory to GPU L2 Cache, the GPU operation bandwidth GPU is obtained. 带宽 ;

[0016] Calculate the number of layers per group for performing inference on the GPU for large models

[0017] Furthermore, each group of model parameters is divided into static parameters and dynamic parameters according to the number of layers K, including:

[0018] Get the parameter size M of one layer of the large model 单层模型参数 ;

[0019] Calculate the parameter size M of the static parameters 静态参数 =(Kt)×M 单层模型参数 , 0 <t<K;

[0020] Calculate the parameter size M of the dynamic parameter 动态参数 =t×M 单层模型参数 ;

[0021] According to the parameter size M of the static parameter 静态参数 and the parameter size M of the dynamic parameters 动态参数 , each group of model parameters is divided into static parameters and dynamic parameters.

[0022] A large model reasoning acceleration system based on dynamic management of model parameters, the system comprising:

[0023] The computing module is used to calculate the number of layers K of each group of inference execution of the large model on the GPU based on the PCIE operating bandwidth and the GPU operating bandwidth;

[0024] A loading module, configured to divide each group of model parameters into static parameters and dynamic parameters according to the number of layers K, load the static parameters into the GPU memory, and save the dynamic parameters in the CPU memory, so as to generate a shared physical memory area of ​​an additional key-value cache and the dynamic parameters in the GPU memory;

[0025] An inference module is used to load the i+1th group of dynamic parameters into the shared physical video memory area through PCIE when executing the i-th group of model inference on the GPU; wherein the model inference time of the i-th group is equivalent to the time of loading the i+1th group of dynamic parameters.

[0026] An electronic device, characterized in that the electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned large model reasoning acceleration methods based on dynamic management of model parameters.

[0027] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the large model reasoning acceleration methods based on dynamic management of model parameters as described above is implemented.

[0028] The technical solution provided by the embodiments of the present disclosure includes at least the following beneficial effects:

[0029] 1) The present invention uses multi-layer parameter dynamic loading technology. The present invention uses a specific arrangement method to load multi-layer model parameters into the same physical location, utilizing idle CPU memory and PCIE bandwidth. This method effectively reduces the video memory usage of model parameters, especially when processing large-scale models, the saved video memory is used to store KV CACHE, thereby increasing the number of batches in the system and achieving the purpose of improving the throughput of the inference system.

[0030] 2) The present invention automatically generates a dynamic video memory loading scheme. The present invention analyzes the PCIE bandwidth and GPU video memory bandwidth of the current hardware, generates a dynamic video memory loading strategy based on the actual available bandwidth, adjusts the additional overhead caused by video memory unloading, and optimizes the efficiency of video memory use. By automatically analyzing and generating a dynamic video memory loading strategy, efficient management and optimization of video memory resources can be achieved, the utilization rate of video memory can be significantly improved, the additional overhead caused by video memory unloading can be reduced, and the overall performance of the system when performing large model reasoning can be improved.

[0031] 3) The inference calculation and dynamic loading of the present invention are pipelined. In the inference calculation process, the present invention pipelines the dynamic parameter loading and calculation process to reduce the additional overhead caused by dynamic loading. By parallelizing the dynamic loading and calculation process, the model inference process can be made more efficient, the waiting time can be reduced, the system response speed can be increased, and the overall inference efficiency can be improved, which helps to achieve faster calculation and response in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram of the interconnection structure between CPU memory and GPU video memory through PCIE.

[0033] Figure 2 Flowchart of the large model inference acceleration method with dynamic video memory management.

[0034] Figure 3 Determine the transmission solution based on the PCIE bandwidth and GPU memory bandwidth.

[0035] Figure 4 Static parameters and dynamic parameters management.

[0036] Figure 5 Inference calculation and parameter dynamic loading overlapping flow chart.

[0037] Figure 6 A throughput comparison chart between the present invention and the prior art. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0039] The large model reasoning acceleration system based on dynamic management of model parameters of the present invention is as follows: Figure 1 As shown, the CPU high-capacity memory is responsible for storing the static parameters of the model and the dynamically loaded parameters, the GPU high-bandwidth video memory is responsible for performing inference calculations, and dynamically loading and unloading model parameters in the video memory, and the PCIE bandwidth realizes fast data transmission between the CPU memory and the GPU video memory. Specifically, the present invention mainly utilizes idle CPU memory and PCIE bandwidth to dynamically load large model parameters to the GPU at runtime, and through specific arrangement, allows multi-layer model parameters to be loaded into the same physical location, reducing the total occupancy of model parameters in the video memory. And through the parallelization of inference calculation and dynamic parameter loading pipelines, the additional overhead caused by dynamically loading models is reduced.

[0040] The method of the present invention for dynamically managing model parameters to accelerate large model reasoning is as follows: Figure 2 As shown, the following steps are included.

[0041] Step 1: Based on the PCIE operating bandwidth and GPU operating bandwidth, calculate the number of layers K for each group of inference execution on the GPU for the large model.

[0042] In one embodiment, the present invention first measures the PCIE operating bandwidth PCIE by measuring the transmission time of 4MB data from the CPU to the GPU. 带宽 , and the GPU bandwidth is measured by measuring the transfer time of 4MB data from GPU memory to GPU L2 Cache 带宽 After that, get the parameter size M of one layer of the model to be loaded 单层模型参数 .

[0043] Afterwards, in order to overlap the loading time of model parameters from CPU to GPU and the inference execution time of K-layer model on GPU (the total number of model layers is N, Group reasoning, K layers of reasoning at a time), such as Figure 3The present invention adjusts the size of the number of groups to make the respective data transmission times equal and avoid idle waiting. According to formula (1), the group size K is evaluated according to the actual measured model parameter size, PCIE bandwidth, and GPU bandwidth (because large model reasoning is mainly limited by memory bandwidth, the actual reasoning time can be approximated by GPU video memory).

[0044]

[0045] Step 2: After dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory and the dynamic parameters are saved in the CPU memory to generate a shared physical memory area for the key-value cache and the dynamic parameters in the GPU memory.

[0046] In the model parameter initialization stage, this embodiment follows The model static parameters are loaded into the video memory in groups, and all parameters that need to be loaded dynamically are stored in the CPU memory. The video memory occupied by the dynamic model parameters is recovered, and the saved video memory is allocated to the KV CACHE, such as Figure 4 As shown in the figure, the static model parameters of layers 1, 2 and layers 4, 5 have been loaded into the GPU memory, while the model parameters of layers 3 and 6 are stored in the CPU memory, saving GPU memory. The saved GPU memory is used for KV CACHE, thereby increasing the number of batches during model inference.

[0047] Step 3: When executing the model inference of the i-th group on the GPU, the i+1-th group of dynamic parameters is loaded into the shared physical video memory area through PCIE; wherein, the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters.

[0048] The present invention performs group model reasoning according to the generation scheme, loads dynamic parameters from the CPU memory to the shared physical video memory in parallel during reasoning, and performs reasoning calculations by group after the parameters required by the current reasoning group are loaded in the video memory. The GPU reasoning of the previous layer and the data transmission from the CPU memory to the GPU video memory of the subsequent layer are synchronized to ensure that the layer parameters required for the next calculation are loaded into the video memory before the next GPU reasoning.

[0049] Specifically, the dynamic parameter loading described in the present invention is to load the dynamic model parameters required for the next group of inference from the CPU memory to the shared physical video memory through the PCIE bandwidth while the GPU inference is being performed. The dynamically loaded parameters are loaded into the same GPU video memory. When performing the i-th group of inference, the calculation time of the K-layer model of the i-th group is equivalent to the time of loading the 1-layer model of the i+1-th group. Therefore, the overhead of model dynamic loading parameters overlaps with the calculation overhead (e.g. Figure 5As shown in Figure 2, multi-layer model parameters are loaded into the same physical location to reduce the occupancy of shared physical video memory.

[0050] Finally, determine whether the reasoning is completed. If not, proceed to the next set of reasoning, otherwise return the reasoning result.

[0051] Below, a specific experiment is used to illustrate the large model reasoning acceleration method based on dynamic management of model parameters provided by the present invention.

[0052] The researchers experimented with the present invention on RTX4050, running the opt1.3b model under limited video memory. The baseline is a method without dynamic video memory management, such as Figure 6 As shown in the figure, 0.23GB of dynamic video memory is saved and used for KVCACHE. When the total memory occupies 3GB, the throughput is increased by 1.41 times.

[0053] In summary, the multi-layer parameter dynamic loading technology of the present invention is to load the multi-layer model parameters into the same physical location on the GPU memory through a specific arrangement method, such as Figure 4 As shown in the figure, layer 3 and layer 6 are dynamically loaded from the CPU memory to the time-sharing shared GPU memory, so as to utilize the idle and large-capacity CPU memory and the idle PCIE bandwidth. When layer 3 and layer 6 that share the GPU memory are requested to access during inference, the parameters are dynamically transferred from the CPU memory to the specified GPU memory through PCIE, thereby improving the utilization of CPU memory and PCIE bandwidth and expanding the memory size that becomes a bottleneck in large model inference.

[0054] The automatic generation of dynamic video memory loading scheme of the present invention is to analyze the PCIE bandwidth and GPU video memory bandwidth (Bandwidth) of the current hardware, and obtain Figure 3 GPU_Memory in Bandwidth (GPU memory bandwidth) and PCIE Bandwidth (PCIE memory bandwidth) These two values, the goal is to adjust Figure 3 The number of inference groups in size K makes Time PCIE (PCIE transmission time) equals Time GPU_Memory (GPU memory transfer time) instead of letting each data load in vain. This generates a dynamic memory loading strategy, reduces the extra overhead of dynamic parameter loading, and optimizes memory usage efficiency.

[0055] The inference calculation and model parameter dynamic loading pipeline of the present invention is to pipeline the dynamic parameter loading and calculation process during the inference calculation process, thereby reducing the additional overhead caused by dynamic loading.

[0056] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A large model reasoning acceleration method based on dynamic management of model parameters, characterized in that: The method comprises: Based on the PCIE operating bandwidth and GPU operating bandwidth, calculate the number of layers K of each group of inference execution on the GPU for the large model; After dividing each group of model parameters into static parameters and dynamic parameters according to the number of layers K, the static parameters are loaded into the GPU memory, and the dynamic parameters are saved in the CPU memory, so as to generate a shared physical memory area of ​​an additional key-value cache and the dynamic parameters in the GPU memory; When executing model inference of the i-th group on the GPU, the i+1-th group of dynamic parameters is loaded into the shared physical video memory area through PCIE; wherein, the model inference time of the i-th group is equivalent to the time of loading the i+1-th group of dynamic parameters.

2. The method according to claim 1, characterized in that The method of calculating the number of layers K of each group of reasoning execution of the large model on the GPU based on the PCIE operation bandwidth and the GPU operation bandwidth includes: According to the data transmission time from CPU to GPU, the PCIE operating bandwidth PCIE is obtained 带宽 ; According to the data transfer time from GPU memory to GPU L2 Cache, the GPU operation bandwidth GPU is obtained. 带宽 ; Calculate the number of layers per group for performing inference on the GPU for large models 3. The method according to claim 1, characterized in that Each group of model parameters is divided into static parameters and dynamic parameters according to the number of layers K, including: Get the parameter size M of one layer of the large model 单层模型参数 ; Calculate the parameter size M of the static parameters 静态参数 =(Kt)×M 单层模型参数 , 0 <t<K; Calculate the parameter size M of the dynamic parameter 动态参数 =t×M 单层模型参数 ; According to the parameter size M of the static parameter 静态参数 and the parameter size M of the dynamic parameters 动态参数 , each group of model parameters is divided into static parameters and dynamic parameters.

4. A large model reasoning acceleration system based on dynamic management of model parameters, characterized in that: The system comprises: The computing module is used to calculate the number of layers K of each group of inference execution of the large model on the GPU based on the PCIE operating bandwidth and the GPU operating bandwidth; A loading module, configured to divide each group of model parameters into static parameters and dynamic parameters according to the number of layers K, load the static parameters into the GPU memory, and save the dynamic parameters in the CPU memory, so as to generate a shared physical memory area of ​​an additional key-value cache and the dynamic parameters in the GPU memory; An inference module is used to load the i+1th group of dynamic parameters into the shared physical video memory area through PCIE when executing the i-th group of model inference on the GPU; wherein the model inference time of the i-th group is equivalent to the time of loading the i+1th group of dynamic parameters.

5. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the large model reasoning acceleration method based on dynamic management of model parameters as described in any one of claims 1-3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the large model reasoning acceleration method based on dynamic management of model parameters as described in any one of claims 1 to 5.

Images